Migrate a Gitlab Docker container from version 8.0.4 to 8.2.0

Update 2016-05-13: The following procedure works perfectly for migrations from version 8.0.4 to 8.7.5

A few days ago, I had to migrate a Gitlab instance installed in Docker from version 8.04 to 8.2.0. I followed the exact steps described in GitLab Docker images documentation. But it was not so easy! Of cource, it required some more! Thus, after 4 hours of errors, searching and trials I found the way! Thank god!

In this article we will make the assumption that Gitlab volumes are stored in /home/bob/docker-data/gitlab directory. Here are the exact steps I followed to make a successful migration:

  1. Stop docker container
    docker stop gitlab
  2. Backup docker volumes (all gitlab files)
    backupDate=$(date +"%Y%m%d%H%M%S") \
      && cd /home/bob/docker-data/ \
      && sudo tar zvcf gitlab-data-${backupDate}.tar.gz gitlab/
  3. Optionally backup docker image
    docker save -o /home/bob/gitlab-ce-image.tar gitlab/gitlab-ce:latest
  4. Remove docker container
    docker rm gitlab
  5. Download the latest Gitlab docker image
    sudo docker pull gitlab/gitlab-ce:latest
  6. After the image is downloaded create and run the container
    docker run -d \
      --hostname \
      --publish 8443:443 --publish 8082:80 --publish 2224:22 \
      --name gitlab \
      --restart always \
      --volume /etc/localtime:/etc/localtime \
      --volume /home/bob/docker-data/gitlab/config:/etc/gitlab \
      --volume /home/bob/docker-data/gitlab/logs:/var/log/gitlab \
      --volume /home/bob/docker-data/gitlab/data:/var/opt/gitlab \
  7. The container starts and we observer the logs via:
    docker logs -f --tail 10 gitlab
  8. The docker container started and we observed in the logs:
    [2015-11-26T15:12:26+02:00] INFO: Retrying execution of execute[create gitlab database user], 19 attempt(s) left
    [2015-11-26T15:12:28+02:00] INFO: Retrying execution of execute[create gitlab database user], 18 attempt(s) left
    ... (some lines omitted) ...
    [2015-11-26T15:13:09+02:00] INFO: Retrying execution of execute[create gitlab database user], 0 attempt(s) left
    Error executing action `run` on resource 'execute[create gitlab database user]'
    Expected process to exit with [0], but received '2'
    ---- Begin output of /opt/gitlab/embedded/bin/psql --port 5432 -h /var/opt/gitlab/postgresql -d template1 -c "CREATE USER gitlab" ----
    STDERR: psql: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
    ---- End output of /opt/gitlab/embedded/bin/psql --port 5432 -h /var/opt/gitlab/postgresql -d template1 -c "CREATE USER gitlab" ----
    Ran /opt/gitlab/embedded/bin/psql --port 5432 -h /var/opt/gitlab/postgresql -d template1 -c "CREATE USER gitlab" returned 2
    Resource Declaration:
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/postgresql.rb
    153:   execute "create #{sql_user} database user" do
    154:     command "#{bin_dir}/psql --port #{pg_port} -h #{postgresql_socket_dir} -d template1 -c \"CREATE USER #{sql_user}\""
    155:     user postgresql_user
    156:     # Added retries to give the service time to start on slower systems
    157:     retries 20
    158:     not_if { !pg_helper.is_running? || pg_helper.user_exists?(sql_user) }
    159:   end
    Compiled Resource:
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/postgresql.rb:153:in `block in from_file'
    execute("create gitlab database user") do
    action [:run]
    retries 20
    retry_delay 2
    default_guard_interpreter :execute
    command "/opt/gitlab/embedded/bin/psql --port 5432 -h /var/opt/gitlab/postgresql -d template1 -c \"CREATE USER gitlab\""
    backup 5
    returns 0
    user "gitlab-psql"
    declared_type :execute
    cookbook_name "gitlab"
    recipe_name "postgresql"
    not_if { #code block }

    These are two known permission issues described in official documentation and in issue #9611. In order to overcome them, we execute:

    docker exec -it gitlab update-permissions
    docker exec gitlab chown -R gitlab-redis /var/opt/gitlab/redis

    and finally we restart the docker container:

    docker restart gitlab
  9. Then we have to check if the database migration was successfull and avoid issue #3255. Thus, we log into bash shell:
    docker exec -t -i gitlab /bin/bash

    and we check the db migration status:

    sudo gitlab-rake db:migrate:status

    If everything is stated as up, we are OK. However, if we notice down migrations like this

    up     20150920161119  Add line code to sent notification
    down    20150924125150  Add project id to ci commit
    down    20150924125436  Migrate project id for ci commits

    we have to rerun the database migration ourserlves:

    sudo gitlab-rake db:migrate

    When it finishes we have to check that everything went ok:

    sudo gitlab-rake db:migrate:status
  10. While being inside the container bash shell, we reconfigure the installation:
    sudo gitlab-ctl reconfigure

    and then check that everything went ok:

    sudo gitlab-rake gitlab:check
  11. If everything is OK, then the command:
    sudo gitlab-rake gitlab:env:info RAILS_ENV=production

    must return something like this:

    System information
    System:   Ubuntu 14.04
    Current User: git
    Using RVM:  no
    Ruby Version: 2.1.7p400
    Gem Version:  2.2.5
    Bundler Version:1.10.6
    Rake Version: 10.4.2
    Sidekiq Version:3.3.0
    GitLab information
    Version:  8.2.0
    Revision: d6bcf44
    Directory:  /opt/gitlab/embedded/service/gitlab-rails
    DB Adapter: postgresql
    HTTP Clone URL:
    SSH Clone URL:  git@
    Using LDAP: yes
    Using Omniauth: no
    GitLab Shell
    Version:  2.6.7
    Repositories: /var/opt/gitlab/git-data/repositories
    Hooks:    /opt/gitlab/embedded/service/gitlab-shell/hooks/
    Git:    /opt/gitlab/embedded/bin/git
  12. Finally we have to clear Redis cache, or we will face the issue #3619 or issue #3609:
    sudo gitlab-rake cache:clear
  13. After that you will be able to login properly to http://localhost:8082


About cmanios

This entry was posted in Docker, Git, Gitlab, Linux, Source control and tagged , , , , , , . Bookmark the permalink.

2 Responses to Migrate a Gitlab Docker container from version 8.0.4 to 8.2.0

  1. Stavros says:

    Dear Cristos,
    Return back to your blog after a long since I found a solution about sms and android database. I am getting emails everytime someone answers to that post, I wanted that.
    I ‘ ve read current post and found very well commented. I just knew github not gitlab. Searching about gitlab I found their serious incident happened couple of months ago. Their full explanation follows https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/
    I am sure you know about it, I just thought is very useful.
    Thank you about usefull tips posts have nice productive time.


    • cmanios says:

      Hello Stavros! Thank you for your kind words! The incident in Gitlab hosted cloud made us aware of how things can go wrong and take more measures in order to avoid it! Keep up the good work!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s