AWS Batch: mount an efs volume into a container using cloud formation

Question

I have an AWS Batch compute environment with a Job Definition.

I created all this using Cloud Formation.

Now I want to add an EFS Volume (Name: EFS-000, File system ID: fs-9999999) and a MountPoint to this Job Definition.

I read

• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html
• https://aws.amazon.com/es/premiumsupport/knowledge-center/batch-mount-efs

In the first link we have an example of a Task Definition ( AWS ECS instead AWS Batch concept)

{
    "containerDefinitions": [
        {
            "memory": 128,
            "portMappings": [
                {
                    "hostPort": 80,
                    "containerPort": 80,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "mountPoints": [
                {
                    "containerPath": "/usr/share/nginx/html",
                    "sourceVolume": "efs-html"
                }
            ],
            "name": "nginx",
            "image": "nginx"
        }
    ],
    "volumes": [
        {
            "name": "efs-html",
            "efsVolumeConfiguration": {
                "fileSystemId": "fs-1324abcd",
                "transitEncryption": "ENABLED"
            }
        }
    ],
    "family": "efs-tutorial"
}

Seems easy add the right code to my Cloud Formation recipe (I choosed yaml syntax). Into my ContainerDefinition I added...

Volumes:
  - Name: SRV 
    EfsVolumeConfiguration:
      FileSystemId: fs-9999999
      TransitEncryption: ENABLED

But when I run the Cloud Formation recipe I get....

The following resource(s) failed to update: [ContentJob1].

Property validation failure: [Encountered unsupported properties in {/ContainerProperties/Volumes/0}: [EfsVolumeConfiguration]]

If EfsVolumeConfiguration is not a valid property...

how I have to do to add an EFS volume to a AWS Batch Job Definition using Cloud Formation?

Answer 1

Until CloudFormation supported EFS, I used this solution. But its still usefull.

I create a Launch Template

  LaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties: 
      LaunchTemplateName: !Join ['', [!Ref ServiceName, '-EC2-', LaunchTemplate]]
      LaunchTemplateData: 
        UserData:
          Fn::Base64: !Sub |
            MIME-Version: 1.0 
            Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

            --==MYBOUNDARY==
            Content-Type: text/cloud-config; charset="us-ascii"

            runcmd:
            - mkdir /mnt/efs-misc
            - mkdir /mnt/efs-jobs
            - echo "${EfsJobs}:/ /mnt/efs-jobs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
            - echo "${EfsMisc}:/ /mnt/efs-misc nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
            - mount -a
            --==MYBOUNDARY==--
        InstanceMarketOptions:
          MarketType: spot
          SpotOptions:
            MaxPrice: 0.096

Then I created a Compute Environment which use this template.

  BatchCompute:
    Type: 'AWS::Batch::ComputeEnvironment'
    DependsOn: LaunchTemplate
    Properties:
      ComputeEnvironmentName: !Join ['', [!Ref ServiceName, '-EC2']]
      ComputeResources:
        AllocationStrategy: SPOT_CAPACITY_OPTIMIZED
        DesiredvCpus: 0
        Ec2KeyPair: !Join ['-', [MyComputeEnv, !Ref EnvironmentLow]]
        InstanceRole: !GetAtt BatchInstanceProfile.Arn
        InstanceTypes:
          - c4.large
          - c5.large
        LaunchTemplate: 
          LaunchTemplateId: !Ref LaunchTemplate
          Version: $Latest
        MaxvCpus: 256
        MinvCpus: 0
        DesiredvCpus: 0
        SecurityGroupIds:
          - !Ref DefaultSecurityGroup
        Subnets: !Split [',', !Join [',', [!Ref SubnetA, !Ref SubnetB, !Ref SubnetC ]]]
        Type: SPOT
        SpotIamFleetRole: SpotFleetTagServiceRole
      ServiceRole: !Ref BatchServiceRole
      State: ENABLED
      Type: Managed

Finaly, I used ContainerProperties/[Volumes|MountPoints] of AWS::Batch::JobDefinition to add my EFS volumes.

  ContentJob0:
    Type: 'AWS::Batch::JobDefinition'
    Properties:
      Type: Container
      ContainerProperties:
        Command: 
          - ls
        Environment:
          - Name: AAA
            Value: AAA
        ExecutionRoleArn: !GetAtt BatchJobRole.Arn
        Image: !Join ['', [!Ref 'AWS::AccountId','.dkr.ecr.', !Ref 'AWS::Region', '.amazonaws.com/', !Ref Image ]]
        LogConfiguration:
          LogDriver: awslogs
        JobRoleArn: !Ref BatchContainerRole
        Memory: 2048
        Vcpus: 2
        Volumes:
          - Name: MISC
            Host:
              SourcePath: '/mnt/efs-0'
          - Name: JOBS
            Host:
              SourcePath: '/mnt/efs-1'
        MountPoints:
          - SourceVolume: MISC
            ContainerPath: '/mnt/efs-0'
          - SourceVolume: JOBS
            ContainerPath: '/mnt/efs-1'
      JobDefinitionName: !Join ['-', ['MyJob', !Ref ServiceName, 'EC2', '0']]
      PlatformCapabilities:
        - EC2
      PropagateTags: true
      RetryStrategy: 
        Attempts: 3
      Timeout: 
        AttemptDurationSeconds: 300

Since I need to set the price of SPOT instances and other thigs into the LaunchTemplate, it had been a good solution to me.

        InstanceMarketOptions:
          MarketType: spot
          SpotOptions:
            MaxPrice: 0.096