AWS Lamda: ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Question

I am new to AWS. I am trying to process the xlsx file using the following AWS services. S3 Bucket, to store the file, SNS to subscribe to AWS Lamda function written in Python. So here is my piece of code:

import json
import boto3
import pandas as pd
import os, tempfile
import sys
import uuid
from urllib.parse import unquote_plus

s3_client = boto3.client('s3')

def lambda_handler(event, context):
    print('coming here')
    message = event['Records'][0]['Sns']['Message']
    bucket = 'bucket1'
    newbucket = 'bucket2'
    jmessage = json.loads(message)
    key = unquote_plus(jmessage["Records"][0]['s3']['object']['key'])
    directory_name = tempfile.mkdtemp()
    download_path = os.path.join(directory_name, 'EXAMPLE2.xlsx')
    print(download_path)
    newkey= 'cleaned.csv'
    upload_path = os.path.join(directory_name, newkey)
    s3_client.download_file(bucket, key, download_path)
    df = pd.read_Excel(download_path, skiprows=3)
    header2 = ['K', 'GEN STATUS']
    df.to_csv(upload_path, columns=header2, index=False)
    s3_client.upload_file(upload_path, newbucket, newkey)

    sns = boto3.client('sns')
    response = sns.publish(
        TopicArn='arn:aws:lambda:us-east-1:<id>:function:DataClean',
        Message='Data is cleaned and save into bucket Cleaned-data. Auto data ingestion is running.'
    )
    return {
        'statusCode': 200,
        'body': json.dumps('Done with cleansing!!')
    }

When I upload the xlsx file to s3 bucket, the SNS triggers the above Lambda function but it renders the following error:

[ERROR] ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden Traceback (most recent call last):
File "/var/task/lambda_function.py", line 23, in lambda_handler s3_client.download_file(bucket, key, download_path)

Here is the complete log:

2021-02-02T12:16:34.838+05:00


[ERROR] ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 23, in lambda_handler
    s3_client.download_file(bucket, key, download_path)
  File "/var/runtime/boto3/s3/inject.py", line 172, in download_file
    extra_args=ExtraArgs, callback=Callback)
  File "/var/runtime/boto3/s3/transfer.py", line 307, in download_file
    future.result()
  File "/var/runtime/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/var/runtime/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/var/runtime/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/var/runtime/s3transfer/download.py", line 343, in _submit
    **transfer_future.meta.call_args.extra_args
  File "/var/runtime/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)

This is what my bucket policy looks like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<id>:root"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::bucket1/*"
        }
    ]
}

So what is going wrong here?

Answer 1

Your lambda role doesnt have access to the S3 bucket to download the objects.

Tutorial: Using AWS Lambda with Amazon S3

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:PutLogEvents",
                "logs:CreateLogGroup",
                "logs:CreateLogStream"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::bucket1/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::bucket2/*"
        }
    ]
}         
        

NOTE: Make sure both(IAM Role and Your S3 Bucket Policy) are allowing your lambda.

Identity-based policies and resource-based policies