Read time: 10 minutes
This post is about attempt to upgrade the GitLab server installation from version 8.x to 10.x. I explained the issues and how I solved them. Tldr? Move to the solution.
If you have GitLab server set up, then visit https://URL_to_your_gitlab/help (your own GitLab server installation). It’ll show some info about your GitLab server like it’s version etc. The server I got my hands on hasn’t been upgraded for a while now. Hence, many packages were out of date. So now while running the “apt upgrade” command, it started to upgrade the packages as usual. It downloaded gitlab-ce latest version 10.x.x as expected. Current version was 8.x.x. So when the package-manager reached to the point of setting up gitlab-ce, it started with the configuration. Everything was going well. Then it got to upgrading Postgresql database server. It also needed an upgrade. So usually it migrates the data/tables as per the new structure introduced in newer version. But while configuring it gave errors.
Checking PostgreSQL executables: OK
Shutting down all GitLab services except those needed for migrations
ok: down: gitlab-workhorse: 4120s, normally up
ok: down: logrotate: 4120s, normally up
ok: down: nginx: 4119s, normally up
ok: down: postgresql: 0s, normally up, want up
ok: down: redis: 0s, normally up
ok: down: sidekiq: 4119s, normally up
ok: down: unicorn: 4119s, normally up
timeout: down: postgresql: 0s, normally up, want up
ok: run: redis: (pid 48594) 0s
down: postgresql: 1s, normally up, want up; run: log: (pid 1075) 38594s
down: postgresql: 1s, normally up, want up; run: log: (pid 1075) 38595s
down: postgresql: 1s, normally up, want up; run: log: (pid 1075) 38596s
down: postgresql: 1s, normally up, want up; run: log: (pid 1075) 38597s
down: postgresql: 1s, normally up, want up; run: log: (pid 1075) 38598s
. . .
and at the end:
Errors were encountered while processing: gitlab-ce
Or even after manually starting postgresql it didn’t show up:
$ sudo gitlab-ctl start postgresql
timeout: down: postgresql: 0s, normally up, want up
$ sudo gitlab-ctl status
run: gitlab-workhorse: (pid 61389) 1255s; run: log: (pid 1073) 50380s
run: logrotate: (pid 61358) 1267s; run: log: (pid 1069) 50380s
run: nginx: (pid 61045) 1481s; run: log: (pid 1076) 50380s
down: postgresql: 0s, normally up, want up; run: log: (pid 64776) 187s
run: redis: (pid 60439) 1730s; run: log: (pid 1071) 50380s
run: sidekiq: (pid 65362) 2s; run: log: (pid 1070) 50380s
run: unicorn: (pid 65335) 10s; run: log: (pid 1068) 50380s
Running sudo gitlab-ctl reconfigure showed this error:
$ sudo gitlab-ctl reconfigure
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.xxxx"?
So it was definitely something wrong with the Postgresql shipped with the gitlab.
I tried to create backup with the following command, but it failed.
$ sudo gitlab-rake gitlab:backup:create
rake aborted!
storage "default" is missing a gitaly_address
While checking the version info, it showed:
$ sudo gitlab-rake gitlab:env:info
rake aborted!
Bundler::GemRequireError: There was an error while trying to load the gem 'uglifier'.
Gem Load Error is: Could not find a JavaScript runtime. See https://github.com/rails/execjs for a list of available runtimes.
I searched about this error and found this and this and got to know that it requires nodejs to be installed now.
$ sudo apt-get install nodejs
After installing nodejs, sudo gitlab-rake gitlab:env:info showed some info, but not about the gitlab yet.
$ sudo gitlab-rake gitlab:env:info
System information
System: Ubuntu 14.04
Current User: git
Using RVM: no
Ruby Version: 2.3.5p376
Gem Version: 2.6.13
Bundler Version:1.13.7
Rake Version: 12.3.0
Redis Version: 3.2.11
Git Version: 2.14.3
Sidekiq Version:5.0.4
Go Version: unknown
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.XXXX"?
One may check about the package version using:
$ sudo apt-cache policy gitlab-ce | grep Installed
Installed: 10.3.3-ce.0
Now the main issue left was that of postgresql. So searched a lot about it too.
In gitlab postgres logs at /var/log/gitlab/postgresql/current, I found entries like this:
LOG: unrecognized configuration parameter "unix_socket_directory" in file "/var/opt/gitlab/postgresql/data/postgresql.conf"
I thought of restoring the backup of the GitLab. GitLab backups are stored at /var/opt/gitlab/backups
I tried the following command for restore. Here 1113118501 is the timestamp of the backup file present at /var/opt/gitlab/backups e.g. 1113118501_gitlab_backup.tar
$ sudo gitlab-rake gitlab:backup:restore BACKUP=1113118501
Unpacking backup ... done
GitLab version mismatch:
Your current GitLab version (10.3.3) differs from the GitLab version in the backup!
Please switch to the following version and try again:
version: 8.6.5
Hint: git checkout v8.6.5
Solution
Then ultimately found this link of an IRC chat on #gitlab channel and this issue. That said it’s 8.x is quite old version and directly upgrading to 10.x would run you into issues.
It said:
Two versions of PG were included in GitLab through the 9.x releases, but we removed the older one in 10.0 since it’s been deprecated for a while.
gableroux try stepping up to 9.x first then 10.x
So for installing 9.x, I went to this link and opened 9.5.9 for my distro:
https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/trusty/gitlab-ce_9.5.9-ce.0_amd64.deb It mentions the steps to install this deb.
The two commands are:
$ curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
Detected operating system as Ubuntu/trusty.
Checking for curl...
Detected curl...
Running apt-get update... done.
Installing apt-transport-https... done.
Installing /etc/apt/sources.list.d/gitlab_gitlab-ce.list...done.
Importing packagecloud gpg key... done.
Running apt-get update... done.
The repository is setup! You can now install packages.
And:
$ sudo apt-get install gitlab-ce=9.5.9-ce.0
dpkg: warning: downgrading gitlab-ce from 10.3.3-ce.0 to 9.5.9-ce.0
(Reading database ... 110272 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_9.5.9-ce.0_amd64.deb ...
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... pg_dump: [archiver (db)] connection to database "gitlabhq_production" failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[FAILED]
Backup failed
gitlab preinstall:
gitlab preinstall: Backup failed! If you want to skip this backup, run the following command and
gitlab preinstall: try again:
gitlab preinstall:
gitlab preinstall: sudo touch /etc/gitlab/skip-auto-migrations
gitlab preinstall:
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_9.5.9-ce.0_amd64.deb (--unpack):
subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
/var/cache/apt/archives/gitlab-ce_9.5.9-ce.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
If you notice in the above error, they suggested to try something. So, as said, I created the file:
$ sudo touch /etc/gitlab/skip-auto-migrations
and tried installing again:
Checking PostgreSQL executables: OK Found /etc/gitlab/skip-auto-migrations, exiting…
This time it skipped the database migrations.
Then I executed:
$ sudo gitlab-ctl reconfigure
Running handlers complete
Chef Client finished, xxx/xxx resources updated in 01 minutes 00 seconds
gitlab Reconfigured!
Restarted gitlab and this time everything was fine.
$ sudo gitlab-ctl restart
ok: run: gitaly: (pid 107106) 1s
ok: run: gitlab-monitor: (pid 107122) 0s
ok: run: gitlab-workhorse: (pid 107125) 1s
ok: run: logrotate: (pid 107132) 0s
ok: run: nginx: (pid 107138) 1s
ok: run: node-exporter: (pid 107147) 0s
ok: run: postgres-exporter: (pid 107187) 0s
ok: run: postgresql: (pid 107195) 0s
ok: run: prometheus: (pid 107203) 1s
ok: run: redis: (pid 107211) 0s
ok: run: redis-exporter: (pid 107215) 0s
ok: run: sidekiq: (pid 107220) 1s
ok: run: unicorn: (pid 107227) 0s
It includes some new modules too, like gitaly. It was missing earlier that’s why it was giving error related to gitaly earlier (mentioned above). But now everything was fine and GitLab was up and running with a new look and feel.
Now time to upgrade to 10.x.
I thought of upgrading postgresql first.
$ sudo gitlab-ctl pg-upgrade
and it completed with the upgrade. Then I executed:
$ sudo apt upgrade
Unpacking gitlab-ce (10.3.4-ce.0) over (9.5.9-ce.0) ...
Upgrade complete! If your GitLab server is misbehaving try running:
sudo gitlab-ctl restart
I restarted as said above some services didn’t start. So I reconfigured again.
$ sudo gitlab-ctl reconfigure
and did some database migrations again. And restarted again and all worked perfectly.
Thanks for reading.