gitlab-runner executor failing on Microk8s Raspberry pi cluster - `ContainersNotInitialized: "containers with incomplete status: [init-permissions]"`

Question

gitlab-runner executor failing on Microk8s Raspberry pi cluster - ContainersNotInitialized: "containers with incomplete status: [init-permissions]"

Any help or just suggestions as to how to troubleshoot this further would be appreciated!

I am trying to us the gitlab-runner executor on a mincrok8s raspberry pi cluster. I am getting the following errors:

Running with gitlab-runner 14.5.2 (e91107dd)
  on gitlab-runner-gitlab-runner-5779968774-dppmf kxEK3YoP
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image arm64v7/ubuntu:20.04 ...
Using attach strategy to execute scripts...
Preparing environment
Waiting for pod gitlab/runner-kxek3yop-project-5-concurrent-0cp2v8 to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab/runner-kxek3yop-project-5-concurrent-0cp2v8 to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab/runner-kxek3yop-project-5-concurrent-0cp2v8 to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
ERROR: Job failed (system failure): prepare environment: waiting for pod running: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

I installed the gitlab-runner executor with helm. Here is the values.yaml I used:

gitlabUrl: http://<my-url-is-here-you-not-need-it>/

runnerRegistrationToken: "xxxxxxxxxxxxxxxx"

concurrent: 20

checkInterval: 30

# For RBAC support:
rbac:
  create: false

runners:
  image: ubuntu:18.04
  helpers:
    image: gitlab/gitlab-runner-helper:arm64-latest

  privileged: false

  builds:
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    cpuRequests: 100m
    memoryRequests: 128Mi

  services:
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    cpuRequests: 100m
    memoryRequests: 128Mi

  helpers:
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    cpuRequests: 100m
    memoryRequests: 128Mi

Answer 1

I had the same issue today and it was beacause of the default helper image which is basically wrong in the chart. Something went really wrong with gitlab's helper image registry because all of the arm images look to be built for amd64! Look here for example: https://hub.docker.com/r/gitlab/gitlab-runner-helper/tags?page=1&name=arm

In any case here's what fixed my problem, in my values.yaml this is how I specified the runners section:

runners:
  tags: "ascalia-k8s"
  secret: gitlab-runner-secret
  namespace: gitlab
  config: |
    [[runners]]
      name = "Kubernetes Prod Runner"
      executor = "kubernetes"
    [runners.kubernetes]
      image = "ubuntu:20.04"
      helper_image = "gitlab/gitlab-runner-helper:arm64-1278d3da"