Lambda returning, 403 Status Code Few times a day on HEAD Operation

Question

I have developed an lambda which fetch the metadata of object stored in S3 bucket and save it to another bucket used to analysis purpose.

In morning, When the Lambda Cold Starts, there is series of failures while fetching, the object Metadata, using Head Operations. The error it return is as follow:

[ERROR] ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Traceback (most recent call last):
  File "/var/task/lumigo_tracer/lambda_tracer/tracer.py", line 55, in lambda_wrapper
    ret_val = func(*args, **kwargs)
  File "/var/task/lumigo_tracer/auto_instrument_handler.py", line 29, in _handler
    return original_handler(*args, **kwargs)
  File "/var/task/handler.py", line 83, in handler
    metadata = s3_client.head_object(Bucket=src_bucket, Key=src_key)["Metadata"]
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)

I did some workaround, and found that the List Operation permission should be provided to lambda, in order to perform this activity. AWS Documentation:

I checked the lambda function it has list operation for required bucket.

The issue is that this error occurs at the time of cold start only.

Time stamp: 6:57:11 AM, 6:58:10 AM and 7:00:21 AM. After that the lambda doesn't observe any failure as mentioned above.

Answer 1

I connected with AWS Support to get the answer, the issue was related with IAM Role, for my lambda.

• I had object level permission for my objects in IAM. This permission is suitable for Head Operation on object.
• The tricky part was some of images were no pushed properly and were deleted by lambda, so it had delete marker instead of object itself. When Lambda request for object it founds the delete marker, and to access the delete marker lambda should have bucket level permissions.
• Thus, this was reason I was getting the 403 error as it required bucket level permissions.

The main outcome was: For object we need object level permissions and for Versions stuff like delete marker, we need bucket level permissions.

The lambda still returns error, but this time it is 404 Object Not found error, which makes sense.

I am posting this answer so that someone can be benefited by this.