Search code examples
dockergitlabupdates

Gitlab upgrade fail and following downgrades fails

I'm running GitLab under Docker with the latest official image.

SPOILER: from my point of view very big mistake using latest if you are a newbie such as I in both: Docker and GitLab as admin!

I updated from version 16.1.4 to 16.4, GitLab could not startup. I saw in the log registry a line that stated that this is a big version jump but no further big error messages from the container manager registry. I saw in the log folder several subfolders with many files, I was not sure what files and folders to check.

I had a bit of panic and...

I decided then to get a fixed image version from the official GitLab Docker hub, while reading in the update path from GitLab I kind of understood that version 16.3 could be a good version to stay at, so I created a container with it, also no luck, GitLab did not start. I did not check in deep the logs, sorry!

I had a bit more panic...

Finally I decided to download the image from my initial version, the 16.1.4, but it does not start neither, probably due to changes in the config file (/etc/gitlab) and more important: changes as well in the database (/var/opt/gitlab)!

Note here that I run the containers pointing to the same /var/opt/gitlab, /etc/gitlab and /var/log/gitlab folders, but those containers have never run in parallel! So MY GUESS at this point is that GitLab updated the files here in the jump from 16.1.4 to 16.4 when I upgraded the container to the 16.4 version so that I will never be able to run properly 16.1.4 and 16.3 because they are older, unless I do a downgrade, if possible.

Now I spent more time on the main Container Manager registry to check the logs, there is a lot of output and for sure I'm missing something, but I see this important line:

# Fatal error loading the DB: Invalid argument. Exiting.
# Can't handle RDB format version 10

I get some info about it and seems I should "update" Redis DB rom the container to the same version GitLab is using... depending on what GitLab version we consider, of course.

Now my dilema is, should I try to sync the Redis DB to the 16.1.4 or should I use 16.4 or 16.3? Also, what is the best way to get started checking the logs from GitLab, is the Container Manager registry enough?

Also I would like to know best practices to deal with Docker and GitLab, all comments are welcome! (such as if it makes sense to go for a fixed image or what kind of updates should I apply to my GitLab, how often and if is better before the updated to create a new container with the updated version or duplicate the existing one and update it with duplicated config and data folders... until now I learnt a lot the hard way... by doing a lot of silly actions :S)

Thanks in advance and sorry for the long story behind!

** EDIT **

I understand now what the GitLab update path is, but in my case, Docker hub offered the latest image so the update path was not the exact recommended one, so what can I do now that all this has happened? Maybe best idea is to start a brand new GitLab container image using 16.4 with blank config and data filders and try to somehow extract/migrate the data from the data backup folder?

Solution

  • So, what I found a GitLab admin should do if s-he was out of the upgrade path that GitLab recommends is to extract the repositories data and install a fresh new instance, this time using a fixed image version, and then upload the projects again.

    Here the bash script to recover the projects I used:

    #!/bin/bash

src=./data/git-data/repositories
dst=./gitlab-repos_bk

for repo in $(printf "%s\n" "$src/@hashed"/*/*/* | grep -Pv '\.wiki\.git$'); do
    # interact with each repo
    path=$(git config --file "$repo/config" --get gitlab.fullpath)
    echo path: $path
    echo repo: $repo
    mkdir -p $dst/$path/.git
    cp -r $repo/* $dst/$path/.git
    git -C $dst/$path --work-tree . reset --hard HEAD
done

    Those are sad news as one expects that such a mature framework like GitLab to apply the updates automatically in the correct order. That's all I could find and as a hobbyist it worked fine for me so I shared the info as it happened, including the long story.

    I would be very much scared to reach such a situation in a production environment.