Search code examples
dockergitlabcontainersgitlab-ci-runner

How is base container executed on Gitlab runner

I am working on a CI/CD pipeline which uses a containerised environment.

  • On my laptop, everything thing is run within a centos container which contains all of my tools, while the code and configuration reside on a shared volume.
  • Most of that shared volume gets pushed up to master when I submit.
    • But, for example, I exclude any python binaries from the virtual environment.
  • That same container is used as my "base" container in Gitlab to execute some of my runner stages in order to
    • execute static lint testing of both the code and the ansible config
    • build the final product as a new image with ansible-bender

In order to distinguish the environments, I use a basic shell in the entrypoint.sh which is executed when the container comes up and in that shell, I used an environment variable which is managed by Gitlab: CI_JOB_STAGE. But the shell script doesn't seem to perform in the same way when I run the container locally as opposed to on the runner. Which is crazy, because that the whole point of containers isn't it?

  • it's unclear at what point Gitlab mounts the share between the runner host and the container
  • conditions testing the value CI_JOB_STAGE don't seem to work even though the values are shown when it printed out with and echo
  • for some reason the entrypoint is executed both before AND after the stage executes

I am attaching gists for

  • the entrypoint.sh script
  • the output when the container is run locally
    • in this case, it see that the virtual env already exist by check for the existence of the activate binary, so that environment is simply activated
    • it then checks for the flask binary to see if it needs to install requirements
    • i know it's primitive, but this is just POC
  • the output for the first stage of the runner
    • based on the shell script, it should not install the requirements because that is part of the stage commands anyway
    • I don't believe it actually does do so, because it goes by too quickly, but the message indicates that it went through the condition
    • if [[ $CI_JOB_STAGE -eq "locally" ]] && ! test -f "./env/bin/flask"
      • if the stage is local (false: we clearly see that stage is lint_code)
      • AND
      • flask is not present (well, it won't be, but it doesn't matter because the first half is already false)

The stages still run correctly, but I find this all very confusing. Does anyone have detailed knowledge of how these runners execute?

gists

Solution

  • Gitlab supports number of executors where each might behave slightly differently (executor might might mount custom volumes, forbid privileged containers)

    The general workflow is:

    1. Prepare: Create and start the services.
    2. Pre-job: Clone, restore cache and download artifacts from previous stages. This is run on a special Docker image.
    3. Job: User build. This is run on the user-provided Docker image.
    4. Post-job: Create cache, upload artifacts to GitLab. This is run on a special Docker Image.

    I would discourage from scripting in entrypoint.sh. try to rewrite it into .gitlab-ci.yml:

    stages:
  - install
  - build
  - test

job 0:
  stage: .pre
  script: install some prerequisites 

job 1:
  stage: install
  script:
    - python3 -m venv env
    - source ./env/bin/activate
    - pip install -r ./dev/requirements.txt

job 2:
  stage: build
  script: make build

job 3:
  stage: test
  script: make test

    It will make builds reproducible and easier to read. You might use .pre and .post stage, just make sure you're using Gitlab >= 12.4.