Search code examples
gitcontinuous-integrationgitlab-cigitlab-ci-runner

Gitlab fetch strategy failed with "could not lock config file"

Today, after a small commit, the gitlab ci pipeline stopped working. It always crashes with an error:

Fetching changes with git depth set to 20...
error: could not lock config file /builds/path/to/project.tmp/git-template/config: File exists

The configuration of the git, runner, server - did not change, it was an ordinary commit, but for some reason it completely broke the pipeline in the entire project.

Now the commit pipelines that worked earlier also don't work. I tried to roll back this commit via hard reset and push --force, but it didn't help.

Only 1 runner is used to run jobs, so I decided to fork the project into the same group and run the pipeline for it... and it works! Moreover, it works with the commit after which the problems began. But the original project still doesn't work.

I also cleaned the runner cache and restarted it (docker executor)

In the end, I saw the git strategy setup in gitlab and put it on git clone (by default git fetch) and wow, the pipeline worked, but when I returned it back to fetch, the same error came out.

I found issues on gitlab with this problem, but either I didn't see a solution, or there is no solution there.

If anyone has any ideas, what can we try to do to make the pipeline work with the 'git fetch` strategy again?

If you need additional information, tell me what - I will update the question

  • Gitlab: 16.0.1-ee free
  • Gitlab runner: 16.0.1

enter image description here

Solution

  • It looks like the problem was solved after restarting the virtual machine with runners.

    A complete list of actions that led to the error disappearing:

    1. Exposed git clone strategy
    2. I started the pipeline - something worked
    3. After a few hours, I rebooted the VM.
    4. I started the pipeline - something worked
    5. Exposed git fetch strategy
    6. The error has disappeared, the pipeline has started working.

    P.s.

    Even on the VM with the runner, I found a directory with git-template containing the config file. He was in /var/lib/docker/volumes/runner-XXXX-cache-YYYY/_data/path/to/project.tmp/git-template/config. Perhaps if you delete all the necessary folders with volumes for runners, the error will also go away