Gitlab runner on aws ec2 : Have one ec2 launching multiple concurrent jobs

Question

I followed this tutorial : https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/

And it works perfectly ! I put the runner in only one branch for testingBut in our pipeline with have like 15 jobs in one stage, but my configuration launch only 2 ec2 machines, so the jobs are taken only 2 at a time, but I want one machine to take 4-5 jobs at the same time

My jobs on the same stage : https://i.sstatic.net/g8Ybz.jpg

This is my config.toml

concurrent = 8
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-runner-xxxxx-dev"
  url = "https://gitlab.com/"
  token = "xxxxxxxxx"
  executor = "docker+machine"
  limit = 2
  request_concurrency = 3
  [runners.docker]
    image = "alpine"
    privileged = true
    disable_cache = true
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      AccessKey = "xxxxxxxx"
      SecretKey = "xxxxx"
      BucketName = "gitlab-runner-xxx-bucket"
      BucketLocation = "eu-west-3"
  [runners.machine]
    IdleCount = 0
    IdleTime = 1800
    MaxBuilds = 10
    MachineDriver = "amazonec2"
    MachineName = "gitlab-docker-machine-%s"
    MachineOptions = [
      "amazonec2-access-key=xxxxxx",
      "amazonec2-secret-key=xxxxxxx",
      "amazonec2-region=eu-west-3",
      "amazonec2-vpc-id=vpc-xxxx",
      "amazonec2-subnet-id=subnet-xxxxx",
      "amazonec2-use-private-address=true",
      "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true",
      "amazonec2-security-group=xxxxx",
      "amazonec2-instance-type=m5.large",
    ]
    [[runners.machine.autoscaling]]
      Periods = ["* * 9-18 * * mon-fri *"]
      IdleCount = 2
      IdleTime = 3600
      Timezone = "UTC"
    [[runners.machine.autoscaling]]
      Periods = ["* * * * * sat,sun *"]
      IdleCount = 1
      IdleTime = 60
      Timezone = "UTC"

Is my config not good for what I want or what do I want is impossible ? Thanks guys !

Answer 1

You've set limit = 2 in your configuration. This will limit the total number of jobs handled by all runners defined in this configuration file to 2.

Set this limit to a higher number to allow more jobs to run concurrently.

See also the relationship between limit, concurrent and IdleCount.