i used to have a always uptodate GitLab instance on my hardware server (Ubuntu 14.04).
However, i decided to install Discourse (or and Docker) lately and now GitLab isnt starting properly anymore.
When i run the service, the sidekiq process gets started but not the Unicorn worker:
git@host:~$ bash -x /etc/init.d/gitlab start
+ RAILS_ENV=production
+ app_user=git
+ app_root=/home/git/gitlab
+ pid_path=/home/git/gitlab/tmp/pids
+ socket_path=/home/git/gitlab/tmp/sockets
+ web_server_pid_path=/home/git/gitlab/tmp/pids/unicorn.pid
+ sidekiq_pid_path=/home/git/gitlab/tmp/pids/sidekiq.pid
+ test -f /etc/default/gitlab
+ . /etc/default/gitlab
++ RAILS_ENV=production
++ app_user=git
++ app_root=/home/git/gitlab
++ pid_path=/home/git/gitlab/tmp/pids
++ socket_path=/home/git/gitlab/tmp/sockets
++ web_server_pid_path=/home/git/gitlab/tmp/pids/unicorn.pid
++ sidekiq_pid_path=/home/git/gitlab/tmp/pids/sidekiq.pid
+ '[' git '!=' git ']'
+ cd /home/git/gitlab
+ check_pids
+ mkdir -p /home/git/gitlab/tmp/pids
+ '[' -f /home/git/gitlab/tmp/pids/unicorn.pid ']'
++ cat /home/git/gitlab/tmp/pids/unicorn.pid
+ wpid=8688
+ '[' -f /home/git/gitlab/tmp/pids/sidekiq.pid ']'
++ cat /home/git/gitlab/tmp/pids/sidekiq.pid
+ spid=2923
+ case "$1" in
+ start_gitlab
+ check_stale_pids
+ check_status
+ check_pids
+ mkdir -p /home/git/gitlab/tmp/pids
+ '[' -f /home/git/gitlab/tmp/pids/unicorn.pid ']'
++ cat /home/git/gitlab/tmp/pids/unicorn.pid
+ wpid=8688
+ '[' -f /home/git/gitlab/tmp/pids/sidekiq.pid ']'
++ cat /home/git/gitlab/tmp/pids/sidekiq.pid
+ spid=2923
+ '[' 8688 -ne 0 ']'
+ kill -0 8688
+ web_status=1
+ '[' 2923 -ne 0 ']'
+ kill -0 2923
+ sidekiq_status=0
+ '[' 1 = 0 -a 0 = 0 ']'
+ gitlab_status=3
+ '[' 8688 '!=' 0 -a 1 '!=' 0 ']'
+ echo 'Removing stale Unicorn web server pid. This is most likely caused by the web server crashing the last time it ran.'
Removing stale Unicorn web server pid. This is most likely caused by the web server crashing the last time it ran.
+ rm /home/git/gitlab/tmp/pids/unicorn.pid
+ '[' 2923 '!=' 0 -a 0 '!=' 0 ']'
+ '[' 1 '!=' 0 -a 0 '!=' 0 ']'
+ '[' 1 '!=' 0 ']'
+ echo -n 'Starting GitLab Unicorn'
Starting GitLab Unicorn+ '[' 1 = 0 ']'
+ rm -f /home/git/gitlab/tmp/sockets/gitlab.socket
+ RAILS_ENV=production
+ bin/web start
master failed to start, check stderr log for details
+ '[' 0 = 0 ']'
+ echo 'The Sidekiq job dispatcher is already running with pid 2923, not restarting'
The Sidekiq job dispatcher is already running with pid 2923, not restarting
+ wait_for_pids
+ i=0
+ '[' '!' -f /home/git/gitlab/tmp/pids/unicorn.pid -o '!' -f /home/git/gitlab/tmp/pids/sidekiq.pid ']'
+ echo
+ print_status
+ check_status
+ check_pids
+ mkdir -p /home/git/gitlab/tmp/pids
+ '[' -f /home/git/gitlab/tmp/pids/unicorn.pid ']'
++ cat /home/git/gitlab/tmp/pids/unicorn.pid
+ wpid=9352
+ '[' -f /home/git/gitlab/tmp/pids/sidekiq.pid ']'
++ cat /home/git/gitlab/tmp/pids/sidekiq.pid
+ spid=2923
+ '[' 9352 -ne 0 ']'
+ kill -0 9352
+ web_status=0
+ '[' 2923 -ne 0 ']'
+ kill -0 2923
+ sidekiq_status=0
+ '[' 0 = 0 -a 0 = 0 ']'
+ gitlab_status=0
+ '[' 0 '!=' 0 -a 0 '!=' 0 ']'
+ '[' 0 = 0 ']'
+ echo 'The GitLab Unicorn web server with pid 9352 is running.'
The GitLab Unicorn web server with pid 9352 is running.
+ '[' 0 = 0 ']'
+ echo 'The GitLab Sidekiq job dispatcher with pid 2923 is running.'
The GitLab Sidekiq job dispatcher with pid 2923 is running.
+ '[' 0 = 0 -a 0 = 0 ']'
+ printf 'GitLab and all its components are \033[32mup and running\033[0m.\n'
GitLab and all its components are up and running.
+ exit
git@host:~$
And no, Docker doesn't try to bind on ports that are used by GitLab:
root@host ~ # netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 3076/redis-server 1
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2940/nginx
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1330/sshd
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 3055/master
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2940/nginx
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1391/mysqld
tcp6 0 0 :::2222 :::* LISTEN 30087/docker-proxy
tcp6 0 0 :::8080 :::* LISTEN 30094/docker-proxy
tcp6 0 0 :::80 :::* LISTEN 2940/nginx
tcp6 0 0 :::4949 :::* LISTEN 1168/perl
tcp6 0 0 :::22 :::* LISTEN 1330/sshd
tcp6 0 0 :::25 :::* LISTEN 3055/master
tcp6 0 0 :::1311 :::* LISTEN 2518/dsm_om_connsvc
udp 0 0 0.0.0.0:51367 0.0.0.0:* 1911/openvpn
Nginx is running, but only shows:
"502 - GitLab is not responding."
Can someone help me out?
Thanks in advance.
I just found the problem.
I somehow overlooked the message in the bash debug output:
master failed to start, check stderr log for details
So I opened /home/git/gitlab/log/unicorn.stderr.log.1 and I saw the following:
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.6.3/lib/unicorn/socket_helper.rb:147:in `initialize': Address already in use - bind(2) for "127.0.0.1" port 8080 (Errno::EADDRINUSE)
And then it hit me: The Nginx of Discourse is using the same port as GitLab's unicorn workers - 8080
I adjusted that port and voilà, everything works again.