I'm executing Nextflow pipelines using AWS Batch.
Managed compute environmentSPOT instancesSPOT_CAPACITY_OPTIMIZED allocation strategy..xlarge to .8xlarge from families of general purpose or memory optimized (e.g. r4., r5. r6i., m4., m5. etc.)As an example, after running the pipeline it is submitting 3 jobs, each of them with defined CPUs 2 and memory 8 GB, but the AWS Batch is deploying for that 3 jobs way larger instance (e.g. r6i.8xlarge, so both memory and CPU are not the bottle neck here), that is being utilized at maybe 20% for all time.
How to set this up, so deployed instances are not permanently underutilized?
I tried to allow smaller instance types, but then jobs are stuck as RUNNABLE and nothing moves for hours.
I tried executing the pipeline with various --max_cpus max_memory but also I don't see any effect here.
What am I doing wrong here?
EDIT:
According to suggestions, I setup three CEs and attached them in the following order to job queue (see config below):
.2xlarge instances, max 64 vCPUs.8xlarge instances, max 128 vCPUsoptimal, max 256 vCPUsI have now 3 jobs stuck in RUNNABLE, each of them with assigned vCPUs 6 and Memory 36864
{
"computeEnvironments": [
{
"computeEnvironmentName": "ce-spot-optimal-spot-capacity-3",
"computeEnvironmentArn": "arn:aws:batch:ap-southeast-1:088159696610:compute-environment/ce-spot-optimal-spot-capacity-3",
"ecsClusterArn": "arn:aws:ecs:ap-southeast-1:088159696610:cluster/AWSBatch-ce-spot-optimal-spot-capacity-3-dbc12b72-6260-315e-a73c-4169455d2a70",
"tags": {},
"type": "MANAGED",
"state": "ENABLED",
"status": "VALID",
"statusReason": "ComputeEnvironment Healthy",
"computeResources": {
"type": "SPOT",
"allocationStrategy": "SPOT_CAPACITY_OPTIMIZED",
"minvCpus": 0,
"maxvCpus": 64,
"desiredvCpus": 24,
"instanceTypes": [
"m4.2xlarge",
"m4.large",
"m4.xlarge",
"m5.2xlarge",
"m5.large",
"m5.xlarge",
"r5.2xlarge",
"r5.large",
"r5.xlarge",
"r6i.2xlarge",
"r6i.large",
"r6i.xlarge"
],
"subnets": [
"subnet-7d67d035",
"subnet-2912954f",
"subnet-c9a4d690"
],
"securityGroupIds": [
"sg-a5c3b2e4"
],
"instanceRole": "arn:aws:iam::088159696610:instance-profile/BM-BatchCEInstanceRole",
"tags": {},
"bidPercentage": 30,
"launchTemplate": {
"launchTemplateName": "increase-volume",
"version": "1"
},
"ec2Configuration": [
{
"imageType": "ECS_AL2",
"imageIdOverride": "ami-0f8ea3f9358cddf80"
}
]
},
"serviceRole": "arn:aws:iam::088159696610:role/aws-service-role/batch.amazonaws.com/AWSServiceRoleForBatch",
"updatePolicy": {
"terminateJobsOnUpdate": false,
"jobExecutionTimeoutMinutes": 30
},
"containerOrchestrationType": "ECS",
"uuid": "5b44dea7-f980-3cd7-92dc-2dc64d0c821c"
},
{
"computeEnvironmentName": "ce-spot-optimal-spot-capacity-2",
"computeEnvironmentArn": "arn:aws:batch:ap-southeast-1:088159696610:compute-environment/ce-spot-optimal-spot-capacity-2",
"ecsClusterArn": "arn:aws:ecs:ap-southeast-1:088159696610:cluster/AWSBatch-ce-spot-optimal-spot-capacity-2-ea6d28fd-495f-34bb-8ea2-1577fc961cf1",
"tags": {},
"type": "MANAGED",
"state": "ENABLED",
"status": "VALID",
"statusReason": "ComputeEnvironment Healthy",
"computeResources": {
"type": "SPOT",
"allocationStrategy": "SPOT_CAPACITY_OPTIMIZED",
"minvCpus": 0,
"maxvCpus": 128,
"desiredvCpus": 0,
"instanceTypes": [
"m4.2xlarge",
"m4.4xlarge",
"m4.large",
"m5.2xlarge",
"m5.4xlarge",
"m5.8xlarge",
"m5.large",
"m5.xlarge",
"r5.2xlarge",
"r5.4xlarge",
"r5.8xlarge",
"r5.large",
"r6i.2xlarge",
"r6i.4xlarge",
"r6i.8xlarge",
"r6i.large",
"m4.xlarge"
],
"subnets": [
"subnet-7d67d035",
"subnet-2912954f",
"subnet-c9a4d690"
],
"securityGroupIds": [
"sg-a5c3b2e4"
],
"instanceRole": "arn:aws:iam::088159696610:instance-profile/BM-BatchCEInstanceRole",
"tags": {},
"bidPercentage": 30,
"launchTemplate": {
"launchTemplateName": "increase-volume",
"version": "1"
},
"ec2Configuration": [
{
"imageType": "ECS_AL2",
"imageIdOverride": "ami-0f8ea3f9358cddf80"
}
]
},
"serviceRole": "arn:aws:iam::088159696610:role/aws-service-role/batch.amazonaws.com/AWSServiceRoleForBatch",
"updatePolicy": {
"terminateJobsOnUpdate": false,
"jobExecutionTimeoutMinutes": 30
},
"containerOrchestrationType": "ECS",
"uuid": "c331302a-8830-3b58-a914-dc54129e2a35"
},
{
"computeEnvironmentName": "ce-spot-optimal-spot-capacity-1",
"computeEnvironmentArn": "arn:aws:batch:ap-southeast-1:088159696610:compute-environment/ce-spot-optimal-spot-capacity-1",
"ecsClusterArn": "arn:aws:ecs:ap-southeast-1:088159696610:cluster/AWSBatch-ce-spot-optimal-spot-capacity-1-6d15c4c4-8f8f-3081-b6af-38f5dfc47fed",
"tags": {},
"type": "MANAGED",
"state": "ENABLED",
"status": "VALID",
"statusReason": "ComputeEnvironment Healthy",
"computeResources": {
"type": "SPOT",
"allocationStrategy": "SPOT_CAPACITY_OPTIMIZED",
"minvCpus": 0,
"maxvCpus": 256,
"desiredvCpus": 0,
"instanceTypes": [
"optimal"
],
"subnets": [
"subnet-7d67d035",
"subnet-2912954f",
"subnet-c9a4d690"
],
"securityGroupIds": [
"sg-a5c3b2e4"
],
"instanceRole": "arn:aws:iam::088159696610:instance-profile/BM-BatchCEInstanceRole",
"tags": {},
"bidPercentage": 30,
"launchTemplate": {
"launchTemplateName": "increase-volume",
"version": "1"
},
"ec2Configuration": [
{
"imageType": "ECS_AL2",
"imageIdOverride": "ami-0f8ea3f9358cddf80"
}
]
},
"serviceRole": "arn:aws:iam::088159696610:role/aws-service-role/batch.amazonaws.com/AWSServiceRoleForBatch",
"updatePolicy": {
"terminateJobsOnUpdate": false,
"jobExecutionTimeoutMinutes": 30
},
"containerOrchestrationType": "ECS",
"uuid": "9a9c493b-4eec-3820-87a8-b86b93ab9341"
}
]
}
Check the order of the compute environments in your job queue. The order of the associated compute environments is used by the scheduler to determine where each of the jobs will be run. So to ensure your smaller jobs are deployed to more appropriate instances, make sure the most appropriate compute environments are listed in ascending order.
Otherwise, I think just using a separate processing queue for the smaller jobs is probably what's needed. A separate queue would let you map up to three compute environments that are more suited for these jobs. You can then assign the job queue to these jobs using the queue directive. This of course can be done using one or more process selectors in your nextflow.config.