ClientError: An error occurred (AccessDenied).. for Step Functions' Lambdas to get_object, despite relevant permissions

Question

As part of an AWS Step Function flow, being run in batches of 20 or more, I have a Lambda (A) that successfully saves a file to s3, then another Lambda (B) that downloads it with boto3 get_object for further processing. The strange thing is; that some of step function iterations fail because the second Lambda (B) runs into a ClientError: An error occurred (AccessDenied), whereas some iterations of the Step function succeed with no errors.

According to AWS; a file saved to s3 should be available immediatly upon saving.

Policy for both Lambdas is this

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Abort*",
                "s3:DeleteObject*",
                "s3:GetBucket*",
                "s3:GetObject*",
                "s3:List*",
                "s3:PutObject",
                "s3:PutObjectLegalHold",
                "s3:PutObjectRetention",
                "s3:PutObjectTagging",
                "s3:PutObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::filesbucket",
                "arn:aws:s3:::filesbucket/*"
            ],
            "Effect": "Allow"
        }
    ]
}

I tried to add a boto3 get_waiter('object_exists') in Lambda (B) but that didn't help and some Step Function iterations failed with Waiter ObjectExists failed: An error occurred (403): Forbidden

Any help would be greatly appreciated.

Answer 1

What I did in the end which worked, was adding retries to the invocation of the second Lambda (B);

Using the CDK;

.add_retry(
        errors=["ClientError"],
        interval=Duration.seconds(4),
        max_attempts=12,
        backoff_rate=1.5,
    )

Or in the State Machine definition;

   "Retry": [
    ........
    {
      "ErrorEquals": [
        "ClientError"
      ],
      "IntervalSeconds": 4,
      "MaxAttempts": 12,
      "BackoffRate": 1.5
    }
  ]

It seems like AWS can take a few seconds for objects saved to S3 to be available for download by boto3.