Search code examples
gitlabgitlab-ci-runnerpodman

Gitlab-runner with podman can't run containers inside properly

Description of the problem:

Hello, We have a pod YAML file with specifications of our containers, their setup, etc. If we run this pod locally with the command 'podman play kube pod.yaml', it starts successfully, and our application runs. But if we run this command inside GitLab-runner (podman as executor), some networking problems occur.

Gitlab runner version: 14.9.2 (CI runner), 15.0.0 (Local GitLab runner)

Description of the problem and differences between running containers in GitLab-runner vs. localhost.

  1. We can't access the external network from the containers inside GitLab-runner (no packages can't be installed etc.). Gitlab-runner container itself can access the external network.
  2. If I try to access POD_NAME:8080/ inside GitLab-runner, it says connection refused. If I try to access POD_NAME:8080/ from localhost with directly running application containers from the host, it says connection refused too.
  3. If I try to access localhost:8080 from the host, it loads the page. If I try to access localhost:8080 from the runner, it says 'No route to host'.
  4. We run the gitlab-runner command with '--privileged' and '--network=host' flags. Also, in our pod, we use 'networkHost: true' and 'privileged: true'.
  5. Containers from GitLab-runner and localhost have different /etc/hosts files:

Containers started from localhost:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.88.0.113 cqe dd88a3440675-infra
127.0.1.1 cqe cqe-dispatcher
127.0.1.1 cqe cqe-umbsender
127.0.1.1 cqe cqe-frontend
127.0.1.1 cqe cqe-db
127.0.1.1 cqe cqe-umbreader

Containers started from runner:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.88.0.117 runner--project-0-concurrent-0 runner--project-0-concurrent-0-d244dbca3614d5aa-build-2
10.88.0.2   cqe 4a46f7216e30-infra
10.88.0.1 host.containers.internal
10.88.0.117 host.containers.internal

Our pod file (I kept only network information and deleted env variables etc.):

apiVersion: v1
kind: Pod
metadata:
  labels:
  name: cqe
spec:
  hostNetwork: true
  privileged: true
  restartPolicy: Always

  containers:
    - name: db
    - name: frontend
      securityContext:
        runAsUser: 5000
        runAsGroup: 5000
      ports:
        - containerPort: 8080
          hostPort: 8080
          protocol: TCP
    - name: dispatcher
      securityContext:
        runAsUser: 5000
        runAsGroup: 5000
    - name: umbreader
      workingDir: /clusterqe-django
      securityContext:
        runAsUser: 5000
        runAsGroup: 5000
    - name: umbsender
      workingDir: /clusterqe-django
      securityContext:
        runAsUser: 5000
        runAsGroup: 5000
  1. Containers have access to each other via all ports. For example, I can access db from the frontend by 'curl POD_NAME:3306'. This also works in containers from GitLab-runner.

I think the problem will probably be related to the fact that we are unpacking a container within a container. But despite using all the different flags and settings, we have not been able to solve this problem for a long time. I will be happy to add more information and reproduction steps.

If I were to describe the main problems that I don't understand:

  1. Containers inside gitlab-runner don't have access to the external network.
  2. I can't access the pod name and its exposed port regardless of whether it's running in the runner or on localhost.
  3. Differences in /etc/hosts and other network settings if I'm inside the runner or on the localhost.
Solution

  • Check if See GitLab 15.1 (June 2022) helps, considering podman is officially supported there.

    GitLab Runner 15.1

    We’re also releasing GitLab Runner 15.1 today!
    GitLab Runner is the lightweight, highly-scalable agent that runs your CI/CD jobs and sends the results back to a GitLab instance.
    GitLab Runner works in conjunction with GitLab CI/CD, the open-source continuous integration service included with GitLab.

    What's new:

    Bug Fixes:

    See Documentation.

    And with GitLab 15.3 (August 2022):

    GitLab Runner 15.3

    We’re also releasing GitLab Runner 15.3 today! GitLab Runner is the lightweight, highly-scalable agent that runs your CI/CD jobs and sends the results back to a GitLab instance. GitLab Runner works in conjunction with GitLab CI/CD, the open-source continuous integration service included with GitLab.

    The list of all changes is in the GitLab Runner CHANGELOG.

    See Documentation.