Received response status [FAILED] from custom resource. Message returned: Command died with <Signals.SIGKILL: 9>

Question

What am I trying to do

• I am using CDK to build a stack that can run a python app
• EC2 to run the python application
• RDS instance to run the PosgreSQL database that connects with EC2
• Custom VPC to contain everything
• I have a local pg_dump of my PostgreSQL database that I want to upload to an S3 bucket which contains all my database data
• I used CDK to create an S3 bucket and tried to upload my pg_dump file

What is happening

• For a small file size < 1MB it seems to work just fine

For my dev dump (About 160 MB in size), it gives me an error

Received response status [FAILED] from
custom resource. Message returned:
Command '['/opt/awscli/aws', 's3',
'cp', 's3://cdk-<some-hash>.zip',
'/tmp/tmpjtgcib_f/<some-hash>']' died
with <Signals.SIGKILL: 9>. (RequestId:
<some-request-id>)

❌  SomeStack failed: Error: The stack
named SomeStack failed creation, it may
need to be manually deleted from the
AWS console: ROLLBACK_COMPLETE:
Received response status [FAILED] from
custom resource. Message returned:
Command '['/opt/awscli/aws', 's3',
'cp', 's3://cdk-<some-hash>.zip',
'/tmp/tmpjtgcib_f/<some-hash>']' died
with <Signals.SIGKILL: 9>. (RequestId:
<some-request-id>)
at
FullCloudFormationDeployment.monitorDeployment

(/Users/vr/.nvm/versions/node/v20.10.0/lib/node_modules/aws-cdk/lib/index.js:455:10568)
at process.processTicksAndRejections
(node:internal/process/task_queues:95:5)
at async Object.deployStack2 [as
deployStack]

(/Users/vr/.nvm/versions/node/v20.10.0/lib/node_modules/aws-cdk/lib/index.js:458:199716)
at async

/Users/vr/.nvm/versions/node/v20.10.0/lib/node_modules/aws-cdk/lib/index.js:458:181438

Code

export class SomeStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // The code that defines your stack goes here

    const dataImportBucket = new s3.Bucket(this, "DataImportBucket", {
      blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
      bucketName: "ch-data-import-bucket",
      encryption: s3.BucketEncryption.KMS_MANAGED,
      enforceSSL: true,
      minimumTLSVersion: 1.2,
      publicReadAccess: false,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      versioned: false,
    });
    
    // This folder will contain my dump file in .tar.gz format
    const dataImportPath = join(__dirname, "..", "assets");

    const deployment = new s3d.BucketDeployment(this, "DatabaseDump", {
      destinationBucket: dataImportBucket,
      extract: true,
      ephemeralStorageSize: cdk.Size.mebibytes(512),
      logRetention: 7,
      memoryLimit: 128,
      retainOnDelete: false,
      sources: [s3d.Source.asset(dataImportPath)],
    });
  }
}

My dev dump file is only about 160 MB but production one is close to a GB. Could someone kindly tell me how I can upload bigger files without this error?

Answer 1

Your lambda is running out of memory. Since it's extracing the .zip archive, it needs to be able to fit its contents in memory. The actual storage size also need to be sufficient - the default is 512 MiB.

Set the memory to e.g. 2 GB and the storage size to the same:

    memoryLimit: 2048,
    ephemeralStorageSize: Size.gibityes(2),