So our GitLab Runner was running fine for a few weeks now. I had to do some upgrades that required a reboot on the machine where it is installed.
After rebooting it no longer picks up jobs automatically. All my pipelines say they're stuck because there are no runners that could pick the jobs up.
I SSH'd into the machine and ran
sudo gitlab-runner run
and the jobs get picked up as I expect them to be. However when I press CTRL + C it stops any picked up jobs and stops picking up any other jobs again. How do I fix this so that it's back to how it was before the reboot?
When I ran
sudo gitlab-runner start
I got an error saying: FATAL: Failed to start gitlab-runner: "systemctl" failed: exit status 5, Failed to start gitlab-runner.service: Unit gitlab-runner.service not found.
A similar error also popped up when I ran
sudo gitlab-runner restart
So from the error message above it's saying the service isn't installed yet (not sure why). Running
gitlab-runner status
does say gitlab-runner: Service is not installed.
So I did
gitlab-runner install -u gitlab-runner
I'm not sure whether gitlab-runner is the default user that was used before this issue came up but I'll just use that here for now at least.
After installing the service I just issued
gitlab-runner start
And the jobs are getting picked up automatically again.
My only guess as to why this happened is because I ran apt-get upgrade which tried to upgrade all components with an update available which of course updated our version of GitLab Runner. Somehow that uninstalls the service? I'm not sure.