I have a Gitlab Runner set up with use docker-machine executor to spin up AWS EC2 instances on-demand.
This is my configuration file:
concurrent = 10
check_interval = 30
[[runners]]
name = "gitlab-runner-manager"
limit = 3
url = "https://git.domain.com"
id = 18
token = "12dcfvgbhnjfcvgbhjnk-9w2"
token_obtained_at = 2023-01-12T06:08:06Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker+machine"
[runners.cache]
Type = "s3"
Shared = true
MaxUploadedArchiveSize = 0
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
BucketName = "domain-gitlab-runner-manager-cache"
BucketLocation = "us-east-1"
[runners.docker]
tls_verify = false
image = "alpine"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = true
shm_size = 0
[runners.machine]
IdleCount = 1
IdleScaleFactor = 0.0
IdleCountMin = 0
IdleTime = 1800
MachineDriver = "amazonec2"
MachineName = "gitlab-runner-autoscale-%s"
MachineOptions = [
"amazonec2-region=us-east-1",
"amazonec2-vpc-id=vpc-123",
"amazonec2-subnet-id=subnet-123",
"amazonec2-iam-instance-profile=gitlab-manager-instance-profile",
"amazonec2-request-spot-instance=true",
"amazonec2-spot-price=0.50",
"amazonec2-use-private-address=true",
"amazonec2-private-address-only=true",
"amazonec2-tags=product,gitlab-runner,environment,shared",
"amazonec2-security-group=gitlab-runner-manager-sg",
"amazonec2-instance-type=t3.large",
"amazonec2-root-size=100",
]
[[runners.machine.autoscaling]]
Periods = ["* * 1-17 * * mon-fri *"]
IdleCount = 1
IdleTime = 3600
Timezone = "UTC"
[[runners.machine.autoscaling]]
Periods = ["* * * * * sat,sun *"]
IdleCount = 0
IdleTime = 3600
Timezone = "UTC"
My servers recently started eroring out with the below snippet from logs:
Feb 14 08:25:21 ip-10-10-0-251.ec2.internal gitlab-runner[2138]: ERROR: Error creating machine: Error running provisioning: Error running "sudo apt-get update": ssh command error: driver=amazonec2 name=runner-nxw1xfxl-gitlab-runner-autoscale-1676363082-24ebaeda operation=create
Feb 14 08:25:21 ip-10-10-0-251.ec2.internal gitlab-runner[2138]: ERROR: command : sudo apt-get update driver=amazonec2 name=runner-nxw1xfxl-gitlab-runner-autoscale-1676363082-24ebaeda operation=create
Feb 14 08:25:21 ip-10-10-0-251.ec2.internal gitlab-runner[2138]: ERROR: err : exit status 100 driver=amazonec2 name=runner-nxw1xfxl-gitlab-runner-autoscale-1676363082-24ebaeda operation=create
Feb 14 08:25:21 ip-10-10-0-251.ec2.internal gitlab-runner[2138]: ERROR: output : Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] driver=amazonec2 name=runner-nxw1xfxl-gitlab-runner-autoscale-1676363082-24ebaeda operation=create
Feb 14 08:25:21 ip-10-10-0-251.ec2.internal gitlab-runner[2138]: ERROR: Reading package lists... driver=amazonec2 name=runner-nxw1xfxl-gitlab-runner-autoscale-1676363082-24ebaeda operation=create
Feb 14 08:25:21 ip-10-10-0-251.ec2.internal gitlab-runner[2138]: ERROR: W: GPG error: http://archive.ubuntu.com/ubuntu focal InRelease: Splitting up /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_InRelease into data and signature failed driver=amazonec2 name=runner-nxw1xfxl-gitlab-runner-autoscale-1676363082-24ebaeda operation=create
Feb 14 08:25:21 ip-10-10-0-251.ec2.internal gitlab-runner[2138]: ERROR: E: The repository 'http://archive.ubuntu.com/ubuntu focal InRelease' is not signed. driver=amazonec2 name=runner-nxw1xfxl-gitlab-runner-autoscale-1676363082-24ebaeda operation=create
The error message is quite straightforward - whereas the sudo apt-get update fails when the EC2 instance is being spun up.
My question now is - how do I sign it to be able to work? I don't even understand what this signing is tbh.
PS: I am not sure if the Gitlab runner part is relavant to the issue but I thought I would share on how I spin up the AWS EC2 instances.
In the end, just added the latest Ubuntu AMI and it works flawlessly. Assuming the default AMI Gitlab uses had some issues. Syntax looks like this (under MachineOptions)
amazonec2-ami=AMI_ID