tls bad certificate with Gitlab Runner on AWS

Question

When starting the gitlab-runner service on an EC2 instance, I am encountering the following:

Nov 10 17:19:04 ip.eu-central-1.compute.internal gitlab-runner[7787]: ERROR: Error creating machine: Error checking the host: Error checking and/or regenerating the certs: There was an error validating certificates for host "ip:2376": remote error: tls: bad certificate  driver=amazonec2 name=runner-XXXXX-gitlab-docker-machine-1699633027-b8326ab1 operation=create

I did not set any certificates manually

My configuration in /etc/gitlab-runner/config.toml is:

concurrent = 10
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "my-test-runner"
  limit = 5
  url = "https://gitlab.com"
  id = 27483251
  token = "XXXXXXXXX"
  token_obtained_at = 2023-11-10T15:19:18Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker+machine"
  [runners.cache]
    Type = "s3"
    Shared = true
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      AccessKey = "XXXXXX"
      SecretKey = "XXXXXX"
      BucketName = "gitlab-runner-s3cache"
      BucketLocation = "eu-central-1"
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = true
    shm_size = 0
    network_mtu = 0
  [runners.machine]
    IdleCount = 2
    IdleScaleFactor = 0.0
    IdleCountMin = 0
    IdleTime = 1800
    MaxBuilds = 100
    MachineDriver = "amazonec2"
    MachineName = "gitlab-docker-machine-%s"
    MachineOptions = ["amazonec2-access-key=XXXXXXXX", "amazonec2-secret-key=XXXXXX", "amazonec2-region=eu-central-1", "amazonec2-vpc-id=vpc-XXXXXXX", "amazonec2-subnet-id=subnet-XXXXX", "amazonec2-use-private-address=true", "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true", "amazonec2-security-group=gitlab-runner-sec-group", "amazonec2-instance-type=t2.small", "amazonec2-request-spot-instance=true", "amazonec2-spot-price=0.05", "amazonec2-zone=a"]

Using root (sudo) to start the gitlab-runner with

sudo systemctl start gitlab-runner

Security group permissions are:

manager-security-group: Outbound all
runner-security-group: Inbound all, Outbound all

[ec2-user@ip ~]$ gitlab-runner --version
Version: 16.5.0

[ec2-user@ip ~]$ sudo docker-machine ls
NAME                                                         ACTIVE   DRIVER      STATE     URL                        SWARM   DOCKER    ERRORS
runner-zx178tugs-gitlab-docker-machine-1699634328-0b78b029   -        amazonec2   Running   tcp://ip:2376           Unknown   Unable to query docker version: Get https://ip:2376/v1.15/version: remote error: tls: bad certificate

Am I missing something? It seems, it should be working.

Answer 1

Solution was to add a docker-machine create before registering the runner in my CloudFormationInit:

"register_gitlab_runner": ec2.InitConfig([
    ec2.InitCommand.shell_command(
        shell_command='docker-machine create --driver none --url localhost dummy-machine',
        key="01_set_docker_machine"
        ),
    ec2.InitCommand.shell_command(
        shell_command=f'sudo gitlab-runner register --template-config "/tmp/config-template.toml" ....',
        key="02_register_gitlab_runner"
    )
])

If you struggle with this problem, here are some (hopefully) helpful links docker-machine with gitlab-runner, change-gitlab-ci-runner-user, docker-machine-dummy