Amazon S3 waitFor() inside AWS Lambda

Question

I'm having issue when calling S3.waitFor() function from inside Lambda function (Serverless nodejs). I'm trying to asynchronously write a file into Amazon S3 using S3.putObject() from one rest api, and poll the result file from another rest api using S3.waitFor() to see if the writing is ready/finished.

Please see the following snippet:

...
S3.waitFor('objectExists', {
  Bucket: bucketName,
  Key: fileName,
  $waiter: {
    maxAttempts: 5,
    delay: 3
  }
}, (error, data) => {
  if (error) {
    console.log("error:" + JSON.stringify(error))
  } else {
    console.log("Success")
  }
});
...

Given valid bucketName and invalid fileName, when the code runs in my local test script, it returns error after 15secs (3secs x 5 retries) and generates result as follows:

error: {
  "message": "Resource is not in the state objectExists",
  "code": "ResourceNotReady",
  "region": null,
  "time": "2018-08-03T06:08:12.276Z",
  "requestId": "AD621033DCEA7670",
  "extendedRequestId": "JNkxddWX3IZfauJJ63SgVwyv5nShQ+Mworb8pgCmb1f/cQbTu3+52aFuEi8XGro72mJ4ik6ZMGA=",
  "retryable": true,
  "statusCode": 404,
  "retryDelay": 3000
}

Meanwhile, when it is running inside AWS lambda function, it returns result directly as follows:

error: {
  "message": "Resource is not in the state objectExists",
  "code": "ResourceNotReady",
  "region": null,
  "time": "2018-08-03T05:49:43.178Z",
  "requestId": "E34D731777472663",
  "extendedRequestId": "ONMGnQkd14gvCfE/FWk54uYRG6Uas/hvV6OYeiax5BTOCVwbxGGvmHxMlOHuHPzxL5gZOahPUGM=",
  "retryable": false,
  "statusCode": 403,
  "retryDelay": 3000
}

As you can see that the retryable and statusCode values are different between the two.

On lamba, it seems that it always get statusCode 403 when the file doesn't exists. While on my local, everything works as expected (retried 5 times every 3 seconds and received statusCode 404).

I wonder if it has anything to do with IAM role. Here's my IAM role statements settings inside my serverless.yml:

iamRoleStatements:
- Effect: "Allow"
 Action:
   - "logs:CreateLogGroup"
   - "logs:CreateLogStream"
   - "logs:PutLogEvents"
   - "ec2:CreateNetworkInterface"
   - "ec2:DescribeNetworkInterfaces"
   - "ec2:DeleteNetworkInterface"
   - "sns:Publish"
   - "sns:Subscribe"
   - "s3:*"
 Resource: "*"

How to make it work from lambda function? Thank you in advance!

Answer 1

It turned out that the key is on how you set the IAM Role for the bucket and all the objects under it.

Based on the AWS docs here, it states that S3.waitFor() is relying on the underlying S3.headObject().

Waits for the objectExists state by periodically calling the underlying S3.headObject() operation every 5 seconds (at most 20 times).

Meanwhile, S3.headObject() itself relies on HEAD Object API which has the following rule as stated on AWS Docs here:

You need the s3:GetObject permission for this operation. For more information, go to Specifying Permissions in a Policy in the Amazon Simple Storage Service Developer Guide. If the object you request does not exist, the error Amazon S3 returns depends on whether you also have the s3:ListBucket permission.

• If you have the s3:ListBucket permission on the bucket, Amazon S3 will return a HTTP status code 404 ("no such key") error.
• if you don’t have the s3:ListBucket permission, Amazon S3 will return a HTTP status code 403 ("access denied") error.

It means that I need to add s3:ListBucket Action to the Bucket resource containing the objects to be able to get response 404 when the objects doesn't exist.

Therefore, I've configured the cloudformation AWS::IAM::Policy resource as below, where I added s3:Get* and s3:List* action specifically on the Bucket itself (i.e.: S3FileStorageBucket).

    "IamPolicyLambdaExecution": {
        "Type": "AWS::IAM::Policy",
        "DependsOn": [
            "IamRoleLambdaExecution",
            "S3FileStorageBucket"
        ],
        "Properties": {
            "PolicyName": { "Fn::Join": ["-", ["Live-RolePolicy", { "Ref": "environment"}]]},
            "PolicyDocument": {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect":"Allow",
                        "Action": [
                            "s3:Get*",
                            "s3:List*"
                        ],
                        "Resource": {
                            "Fn::Join": [
                                "",
                                [
                                    "arn:aws:s3:::",
                                    {
                                        "Ref": "S3FileStorageBucket"
                                    }
                                ]
                            ]
                        }
                    },
                    {
                        "Effect":"Allow",
                        "Action": [
                            "s3:GetObject",
                            "s3:PutObject",
                            "s3:DeleteObject"
                        ],
                        "Resource": {
                            "Fn::Join": [
                                "",
                                [
                                    "arn:aws:s3:::",
                                    {
                                        "Ref": "S3FileStorageBucket"
                                    },
                                    "/*"
                                ]
                            ]
                        }
                    },
                    ...

Now I've been able to do S3.waitFor() to poll for file/object under the bucket with only a single API call and get the result only when it's ready, or throw error when the resource is not ready after a specific timeout.

That way, the client implementation will be much simpler. As it doesn't have to implement poll by itself.

Hope that someone find it usefull. Thank you.