So after finally getting GitLab set up and using it for the day, I decide to attempt to enable the GitLab Registry. However, after running gitlab-ctl reconfigure the system enters a boot-loop (every boot lasts for 10-15s).
Now I've been searching everywhere for ways to sort this out, end the boot-loop or get the logs that detail why the container keeps rebooting.
I'm running the Omnibus Docker image of GitLab (latest version) on a Digital Ocean droplet VPS. GitLab is configured to not use https as I use a reverse proxy (apache) to connect (this connection is over https).
Reverse Proxy configuration Browser => example.com:443 (https) => localhost:8888 (http)
Docker ps output
c13e26a20f7d gitlab/gitlab-ce:latest "/assets/wrapper" 24 hours ago Up 2 seconds (health: starting) 0.0.0.0:2222->22/tcp, 0.0.0.0:8888->80/tcp, 0.0.0.0:4444->443/tcp gitlab
What I have tried (with no success):
Any suggestions and/or how I would go about fixing this issue would be greatly appreciated!
Fix: Shutting down the Docker Container, inspecting the reconfiguration logs (located in the logs/reconfiguration directory inside the installation folder) and making adjustments to the configuration file based on the errors apparent in the log.
Fixing the boot-loop (step by step)
NOTE: If you are not running as root, I recommend temporarily running as root (using sudo su) to avoid Permission denied errors.
After manually stopping the Docker Container I was able to view the full log located at /srv/gitlab/logs/reconfigure (with the latest timestamp)
Excerpt from the latest reconfiguration log:
[2018-05-19T00:24:50+00:00] INFO: *** Chef 13.6.4 ***
[2018-05-19T00:24:50+00:00] INFO: Platform: x86_64-linux
[2018-05-19T00:24:50+00:00] INFO: Chef-client pid: 25
[2018-05-19T00:24:50+00:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2018-05-19T00:24:52+00:00] WARN: Plugin Network: unable to detect ipaddress
[2018-05-19T00:24:52+00:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
[2018-05-19T00:24:52+00:00] INFO: Run List is [recipe[gitlab]]
[2018-05-19T00:24:52+00:00] INFO: Run List expands to [gitlab]
[2018-05-19T00:24:52+00:00] INFO: Starting Chef Run for [REDACTED]
[2018-05-19T00:24:52+00:00] INFO: Running start handlers
[2018-05-19T00:24:52+00:00] INFO: Start handlers complete.
[2018-05-19T00:24:54+00:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, postgresql@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.0.0, gitaly@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@0.14.2, acme@3.1.0, crond@0.1.0, compat_resource@12.19.0]
[2018-05-19T00:24:56+00:00] WARN: Runtime directory '/run' is not a tmpfs.
Error messages are a good start, but won't reveal the underlying issue...
[2018-05-19T00:24:56+00:00] ERROR: Running exception handlers
[2018-05-19T00:24:56+00:00] ERROR: Exception handlers complete
Now the last 3 lines of the log revealed the problem
[2018-05-19T00:24:56+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2018-05-19T00:24:56+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2018-05-19T00:24:56+00:00] FATAL: RuntimeError: Unsupported GitLab Registry external URL path: /gitlab/registry
Now after seeing the very last line, I see that my configuration is invalid and is the cause of the fatal issue.
To fix the issue, I edited the configuration while the docker container was in an unbooted state (configuration file located at /srv/gitlab/config/gitlab.rb). After fixing the configuration, which in my case was to comment out all GitLab Registry configuration options (since I decided to wait with testing this feature for now).
Diff for configuration
################################################################################
## Container Registry settings
##! Docs: https://docs.gitlab.com/ce/administration/container_registry.html
################################################################################
- registry_external_url 'http://[REDACTED]:4567/gitlab/registry'
+ # registry_external_url 'http://[REDACTED]:4567/gitlab/registry'
### Settings used by GitLab application
- gitlab_rails['registry_enabled'] = true
+ # gitlab_rails['registry_enabled'] = true
- gitlab_rails['registry_host'] = "[REDACTED]"
+ # gitlab_rails['registry_host'] = "[REDACTED]"
- gitlab_rails['registry_port'] = "4567"
+ # gitlab_rails['registry_port'] = "4567"
- gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
+ # gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
I ran the following commands to boot the container and apply the new configuration.
docker start gitlab
sudo docker exec gitlab gitlab-ctl reconfigure
Promising results right away as the Container did not reboot right after I issued the command.
Starting Chef Client, version 13.6.4
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
- gitlab (0.0.1)
- package (0.1.0)
- registry (0.1.0)
- postgresql (0.1.0)
- letsencrypt (0.1.0)
- mattermost (0.1.0)
- runit (0.14.2)
- nginx (0.1.0)
- gitaly (0.1.0)
- consul (0.0.0)
- acme (3.1.0)
- crond (0.1.0)
- compat_resource (12.19.0)
Installing Cookbook Gems:
...
That's it! Now everything works as it should. Turned out user error was the cause of the issue.