Upgrade GitLab to v17.4.1 #3720

SilinPavel · 2024-09-27T12:15:04Z

Background
It would be great to have possibility to deploy CloudPipeline with fresh GitLab version (17.4.1 as of now)

SilinPavel · 2024-09-30T12:15:19Z

GitLab upgrade from v15.5.4 to 17.4.1

Initial setup:

postgres:12.4
gitlab:15.5.4

Based on :

Upgrade path tool: https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/?current=15.5.0&edition=ce
PostgreSQL matrix: https://docs.gitlab.com/ee/administration/package_information/postgresql_versions.html
Installation reqs: https://docs.gitlab.com/ee/install/requirements.html#postgresql

Update process will look as follow:

Uprade steps

Backup current stage
Restore backup to the temp GitLab deployment
-> postgres:14.11
-> Migrate PAC tokens if needed
-> gitlab:15.11.13
-> gitlab:16.3.8
-> gitlab:16.7.9
-> gitlab:16.11.10
-> gitlab:17.3.3
-> gitlab:17.4.1
Create backup and migrate it to the kubernetes GitLab deployment

0. Backup current stage

Create backup:

kubectl exec -it <gitlab gitlab-ctl stop

gitlab-backup create

kubectl exec -it <gitlab gitlab-ctl start

gitlab_settings_bkp_file="/var/opt/gitlab/backups/settings-dump-$(date +%Y%m%d).tgz"
tar -czf $gitlab_settings_bkp_file /etc/gitlab

Save backup from /var/opt/gitlab/backups/ somewhere save and accessable

The following steps will restore created backup into temp gitlab deployment, then will update this deployment up to the latest version.
Finally, we will create another backup for the final version and as a last step - restore this backup to the kubernetes gitlab deployment.

1. Restore backup to the temp GitLab deployment

Before upgrade let's prepare environment. First of all, create working directories:

mkdir $GITLAB_HOME/logs
mkdir $GITLAB_HOME/config
mkdir $GITLAB_HOME/data

Create gitlab.rb file into config directory with the following content:

gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
gitlab_rails['db_host'] = 'localhost'
gitlab_rails['db_port'] = <DB port>
gitlab_rails['db_username'] = 'gitlab'
gitlab_rails['db_password'] = 'gitlab'
postgresql['enable'] = false

Let's prepare database:

docker run --name postgres12 -e POSTGRES_PASSWORD=postgres -d -p <DB port>:5432 postgres:12.4
docker exec postgres9 psql -U postgres -c "create user gitlab createdb;"
docker exec postgres9 psql -U postgres -c "alter user gitlab with superuser;"
docker exec postgres9 psql -U postgres -c "alter user gitlab with password 'gitlab';"
docker exec postgres9 psql -U postgres -c "create database gitlabhq_production owner gitlab;"

Now we are ready to run gitlab. This gitlab will be empty and connected to database refined below.

The version of gitlab shall be exactly the same as gitlab version from backup dumb.
docker run -d --network=host \
  --hostname gitlab.example.com \
  --name gitlab \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ce:12.5.4-ce.0

Once gitlab is ready and healthy:

stop gitlab (docker stop gitlab)
copy etc/gitlab from cp-git-settings backup file to $GITLAB_HOME/config (except for gitlab.rb file)
place backup dumb into $GITLAB_HOME/data/backups.
Start gitlab again: docker restart gitlab

Stop processes that may have connection with database and restore backup:

docker exec gitlab gitlab-ctl stop puma
docker exec gitlab gitlab-ctl stop sidekiq
# Verify
docker exec gitlab gitlab-ctl status
# Restore
docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=<backup timestamp>

Answer yes for the first question:

Before restoring the database we recommend removing all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.
 
Do you want to continue (yes/no)? yes

Answer no for the second:

This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.
Do you want to continue (yes/no)? no

Restart gitlab container and wait for healthy state:

docker restart gitlab
# Check state
docker exec -it gitlab gitlab-rake gitlab:check SANITIZE=true

Now the gitlab copy is ready for upgrade.

2. Upgrade to postgres:14.11

Let's prepare database:

docker run --name postgres14 -e POSTGRES_PASSWORD=postgres -d -p <DB port>:5432 postgres:14.11
docker exec postgres9 psql -U postgres -c "create user gitlab createdb;"
docker exec postgres9 psql -U postgres -c "alter user gitlab with superuser;"
docker exec postgres9 psql -U postgres -c "alter user gitlab with password 'gitlab';"
docker exec postgres9 psql -U postgres -c "create database gitlabhq_production owner gitlab;"

Once PostgreSQL container prepared make a database backup:

Stop processes that are connected to DB
docker exec gitlab gitlab-ctl stop puma
docker exec gitlab gitlab-ctl stop sidekiq
# Verify
docker exec gitlab gitlab-ctl status

Update $GITLAB_HOME/config/gitlab.rb file: postgresql['version'] = 14

Create database backup:

docker exec gitlab gitlab-ctl reconfigure
docker exec gitlab gitlab-ctl stop

# Create database backup and make a note of the backup file name <backup timestamp>
docker exec gitlab gitlab-rake gitlab:backup:create SKIP=repositories,uploads,builds,artifacts,lfs,pages,registry

Update $GITLAB_HOME/config/gitlab.rb file with a new DB configurations (to connect to the new db instance)

Restore backup:

NOTE: is not a file name. This is a part before _gitlab_backup.tar . For example, if we have a backup file with name 1666196708_2022_10_19_12.0.12_gitlab_backup.tar the in this case will be 1666196708_2022_10_19_12.0.12 and the command will be:

gitlab-rake gitlab:backup:restore BACKUP=/var/opt/gitlab/backups/1666196708_2022_10_19_12.0.12 
docker exec gitlab gitlab-ctl reconfigure

# Restore database backup:
docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=<backup timestamp>

Answer yes for the first question:

Before restoring the database we recommend removing all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.

Do you want to continue (yes/no)? yes

Answer no for the second:

This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.
Do you want to continue (yes/no)? no

docker exec gitlab gitlab-ctl start

wait for gitlab api up and check everything is ok: repositories, commit availability, etc

3. Migrate PAC tokens:

Previous deployment db can have problematic PAC tokens, which will lead to the fail of the update of GitLab.
To fix this, connect to the GitLab db and run the following SQL update query:

update personal_access_tokens set scopes = REPLACE(scopes, ' !ruby/array:Hashie::Array', '')

4. Upgrade to gitlab:15.11.13

How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \
  --hostname gitlab.example.com \
  --name gitlab \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ce:15.11.13-ce.0

Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

5. Upgrade to gitlab:16.3.8

How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \
  --hostname gitlab.example.com \
  --name gitlab \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ce:16.3.8-ce.0

Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

6. Upgrade to gitlab:16.7.9

How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \
  --hostname gitlab.example.com \
  --name gitlab \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ce:16.7.9-ce.0

Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

7. Upgrade to gitlab:16.11.10

How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \
  --hostname gitlab.example.com \
  --name gitlab \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ce:16.11.10-ce.0

Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

8. Upgrade to gitlab:17.3.3

How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \
  --hostname gitlab.example.com \
  --name gitlab \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ce:17.3.3-ce.0

Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

9. Upgrade to gitlab:17.4.1

How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \
  --hostname gitlab.example.com \
  --name gitlab \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ce:17.4.1-ce.0

Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

10. Create backup and migrate it to the kubernetes GitLab deployment

Create backup

Create backup:

gitlab-backup create

gitlab_settings_bkp_file="/var/opt/gitlab/backups/settings-dump-$(date +%Y%m%d).tgz"
tar -czf $gitlab_settings_bkp_file /etc/gitlab

Save backup from /var/opt/gitlab/backups/ (host path: $GITLAB_HOME/data/backups/) somewhere save and accessable

Deploy new cp-gitlab-db pod

Create a new folder for gitlab db on the system share or local folder on a mater node which hosts all deployment dirs, f.e:
```
mkdir /opt/gitlab-17-postgresql/data
```
Using templates deploy/contents/k8s/cp-gitlab-db/cp-gitlab-db-dpl.yaml and deploy/contents/k8s/cp-gitlab-db/cp-gitlab-db-svc.yaml create fresh gitlab-db deployment
- Change labels, selectors and names to be identical (f.e. cp-gitlab-db -> cp-gitlab-db-17)
- Change volumes/gitlab-pgdata from /opt/gitlab-postgresql/data to /opt/gitlab-17-postgresql/data
- Apply this configs with kubectl apply -f <file>

Create empty gitlab db:

Inside the gitlab-db container execute:

psql -U postgres -c "create user gitlab createdb;"
psql -U postgres -c "alter user gitlab with superuser;"
psql -U postgres -c "alter user gitlab with password 'gitlab';"
psql -U postgres -c "create database gitlabhq_production owner gitlab;"

Deploy new cp-gitlab pod

Create a new folder for gitlab on the system share or local folder on a mater node which hosts all deployment dirs, f.e:

mkdir -p /opt/gitlab-17/logs
mkdir -p /opt/gitlab-17/config
mkdir -p /opt/gitlab-17/data
mkdir -p /opt/gitlab-17/pki
cp -r /opt/gitlab/pki/* /opt/gitlab-17/pki/

Using templates deploy/contents/k8s/cp-git/cp-git-dpl.yaml create fresh gitlab deployment
- Change labels, selectors and names to be identical (f.e. cp-git -> cp-git-17)
- Change volumes/git-logs from /opt/gitlab/logsto/opt/gitlab-17/logs. The same for volumes volumes/git-config, volumes/git-data volumes/git-pki`
- Override gitlab-db connection with containers/env section:
```
env:
- name: GITLAB_DATABASE_HOST
  value: <cp-gitlab-db-17 svc address> f.e.: cp-gitlab-db-17.default.svc.cluster.local
- name: GITLAB_DATABASE_PORT
  value: "6543"
- name: GITLAB_ROOT_PASSWORD
  value: <gitlab password>
```
- Apply this configs with kubectl apply -f <file>

After appliyng:

Copy prepared backup

cp <backup location>/<backup>_gitlab_backup.tar /opt/gitlab/data/backups/
chmod a+r /opt/gitlab-17/data/backups/<backup>_gitlab_backup.tar

Restore backup

kubectl exec -it $GITLAB_POD gitlab-ctl stop puma
kubectl exec -it $GITLAB_POD gitlab-ctl stop sidekiq
kubectl exec -it $GITLAB_POD gitlab-rake gitlab:backup:restore BACKUP=$GITLAB_BACKUP
--- yes
--- no

Copy configuration files from settings-dump backup (except for gitlab.rb) "cp /etc/* /opt/gitlab/config/"

Reconfigure gitlab

kubectl exec -it $GITLAB_POD gitlab-ctl reconfigure

Restart git pod and check status
kubectl exec -it $GITLAB_POD gitlab-rake gitlab:check SANITIZE=true

Things to consider:

Starting from GitLab 16, all tokens will be < 365 days lifetime, which means

SilinPavel added the kind/enhancement New feature or request label Sep 27, 2024

SilinPavel self-assigned this Sep 27, 2024

SilinPavel mentioned this issue Sep 27, 2024

Issue 3720 update install procedure to be able to work with GitLab v17 #3721

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade GitLab to v17.4.1 #3720

Upgrade GitLab to v17.4.1 #3720

SilinPavel commented Sep 27, 2024

SilinPavel commented Sep 30, 2024

Upgrade GitLab to v17.4.1 #3720

Upgrade GitLab to v17.4.1 #3720

Comments

SilinPavel commented Sep 27, 2024

SilinPavel commented Sep 30, 2024

GitLab upgrade from v15.5.4 to 17.4.1

Uprade steps

0. Backup current stage

1. Restore backup to the temp GitLab deployment

2. Upgrade to postgres:14.11

3. Migrate PAC tokens:

4. Upgrade to gitlab:15.11.13

How to upgrade gitlab instalce

5. Upgrade to gitlab:16.3.8

How to upgrade gitlab instalce

6. Upgrade to gitlab:16.7.9

How to upgrade gitlab instalce

7. Upgrade to gitlab:16.11.10

How to upgrade gitlab instalce

8. Upgrade to gitlab:17.3.3

How to upgrade gitlab instalce

9. Upgrade to gitlab:17.4.1

How to upgrade gitlab instalce

10. Create backup and migrate it to the kubernetes GitLab deployment

Create backup

Deploy new cp-gitlab-db pod

Create empty gitlab db:

Deploy new cp-gitlab pod

Things to consider: