Search code examples
gitlab-ci-runnersystemdcgroups

How can I assign cgroup resource limits per gitlab runner job to ensure fair sharing of resources?

I'd like to assign cgroup limits to my gitlab runner jobs so that they each get a fair amount of CPU and IO.

I've tried using sudo cgclassify to a slot specific cgroup within the job scripts, but the subprocesses of the job don't seem to inherit the same cgroup nor the limits. They reset back to the gitlab runner service's cgroup and its limits. I think systemd is monitoring future forks and assigning the cgroup.

Solution

  • What seems to work is to not try to take the job out of its own cgroup / scope with cgclassify. It turns out that each gitlab runner job is already put into its own systemd scope. So we just need to modify the limits in that scope.

    This script will determine what scope you are in and apply the limit. If call this from your gitlab cicd script with sudo, you can assign the resource limits:

    #!/bin/sh
set -x
set -e
echo CGROUP Before:
cat /proc/self/cgroup
scope=$(grep 1:name=systemd: /proc/self/cgroup | grep -o -E 'session-\S+.scope')
systemctl set-property --runtime $scope CPUShares=1000 BlockIOWeight=500
echo CGROUP After:
cat /proc/self/cgroup

    And the sudoers ...

    gitlab-runner ALL=(root) NOPASSWD: /usr/local/bin/limit-gitlab-job-resources