How to run gitlab-runner with local cache under kubernetes

Question

I'm trying to use a local cache on gitlab-runners under k8s - I do not want to use s3.

I'm reading the docs over at https://docs.gitlab.com/runner/executors/kubernetes.html#using-the-cache-with-the-kubernetes-executor:

When the cache is used with the Kubernetes executor, a volume called /cache is mounted on the pod. During job execution, if cached data is needed, the runner checks if cached data is available. Cached data is available if a compressed file is available on the cache volume.

To set the cache volume, use the cache_dir setting in the config.toml file.

If available, the compressed file is extracted into the build folder and can then be used in the job. If not available, the cached data is downloaded from the configured storage and saved into the cache dir as a compressed file. The compressed file is then extracted into the build folder.

This together with the description of how the runner is starting executor pods for each job by the k8s API, I got the sense the cache would just work. The runners themselves are long-lived in my setup - no autoscaling and it would be good enough if the cache lived in them and was auto-mounted into the executor pods.

It doesn't seem to work though and the logs are confusing.

This is my test pipeline:

default:
  cache: &cache
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - local-cached-folder

prepare: 
  stage: prepare
  cache:
    <<: *cache
    policy: pull-push
  script:
    - mkdir local-cached-folder
    - echo "preparing…" > ./local-cached-folder/cached_file
    - echo "ls /cache"
    - ls -lFa /cache || true

build:
  stage: build
  cache:
    <<: *cache
    policy: pull-push
  script:
    - echo "ls -lFa ."
    - ls -lFa .
    - echo "ls -lFa local-cached-folder"
    - ls -lFa local-cached-folder || true
    - echo "ls /cache"
    - ls -lFa /cache || true

What's weird is that the logs are saying "Successfully extracted cache", but then there's nothing there:

Created fresh repository.
Checking out 7e436847 as detached HEAD (ref is master)...
Skipping Git submodules setup
Restoring cache
00:01
Checking cache for master-6-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
00:01
$ mkdir local-cached-folder
$ echo "preparing…" > ./local-cached-folder/cached_file
$ echo "ls /cache"
ls /cache
$ ls -lFa /cache || true
ls: cannot access '/cache': No such file or directory
Saving cache for successful job
00:01
Creating cache master-6-protected...
local-cached-folder: found 2 matching artifact files and directories 
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally. 
Created cache

And on the next job:

Restoring cache
00:01
Checking cache for master-6-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
00:01
$ echo "ls -lFa ."
ls -lFa .
$ ls -lFa .
…
$ echo "ls -lFa local-cached-folder"
ls -lFa local-cached-folder
$ ls -lFa local-cached-folder || true
ls: cannot access 'local-cached-folder': No such file or directory
$ echo "ls /cache"
ls /cache
$ ls -lFa /cache || true
ls: cannot access '/cache': No such file or directory
Saving cache for successful job
00:01
Creating cache master-6-protected...
WARNING: local-cached-folder: no matching files. Ensure that the artifact path is relative to the working directory (/builds/kostbevakningen/kb-frontend) 

To me it sounds like it finds a cache archive, but maybe this is a combination of me misunderstanding the docs and bad logging?

I did read gitlab-runner kubernetes cache is ignored and figured I should probably mount a PVC, using [[runners.kubernetes.volumes.pvc]] but it's pretty old and the accepted answer just says S3.

Answer 1

I got it working by using a PVC on the executor by putting it in the toml in the chart. My full values.yaml for the official helm chart is now:

gitlabUrl: https://gitlab.com/
rbac:
  create: true

runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "ubuntu:22.04"
        helper_image = "gitlab/gitlab-runner-helper:alpine-latest-arm64-v16.1.0"

      [[runners.kubernetes.volumes.pvc]]
        name = "runners-cache"
        mount_path = "/cache"
  executor: kubernetes

volumeMounts: []
volumes: []

So it seems /cache is the default and something needs to be mounted there. The logs that said "Successfully extracted cache" seems like total nonsense.