Extract JSON-like information from payloads produced by AWS SNS

Question

I'm currently working on the implementation of SNS-notifications being the intermediary between S3-bucket-uploads and an upload-handler Lambda Function.

The information flow should look like this:

1. Upload of a file to an S3-bucket
2. This should trigger an SNS-topic "upload-notification"
3. Lambda-function ("upload-handler") are subscribed to this SNS-topic: depending on which bucket received a file upload, a certain lambda-function should be triggered by the SNS-notification

--> How can I obtain information like the "S3 bucket name" etc. from the event that triggered the SNS-notification?

I hope for a possibility like with lambda functions where you can extract information e.g. from a JSON-object produced by SNS. If that doesn't exist, I'd be delighted to learn about other approaches, but somehow I need to extract this information programmatically/automatically from SNS to hand it over to the upload-handler lambda function in Step 3.

---

Details on the terraform definition blocks:

1. aws_sns_topic_subscription:

resource "aws_sns_topic_subscription" "start_from_upload_topic" {
  topic_arn = var.upload_notification_topic_arn
  protocol  = "lambda"
  endpoint  = module.start_from_upload_handler.arn
}

2. aws_s3_bucket_notification:

resource "aws_s3_bucket_notification" "start_from_upload_handler" {
  for_each = local.input_bucket_id_map

  bucket = each.value
  topic {
    topic_arn = module.upload_notification.topic_setup.topic_arn
    events    = ["s3:ObjectCreated:*"]
  }
}

3. SNS-module "upload_notification"

module "upload_notification" {
  source  = "../../modules/sns_topic"
  name    = "${var.platform_settings.prefix}-upload-notification"
  key_arn = var.platform_settings.logging_settings.logging_key_arn
  allowed_producers = [
    "s3.amazonaws.com",
    "lambda.amazonaws.com",
    "edgelambda.amazonaws.com",
    "events.amazonaws.com",
    "states.amazonaws.com",
  ]
  allowed_consumers = ["lambda.amazonaws.com",
    "edgelambda.amazonaws.com",
    "events.amazonaws.com",
    "states.amazonaws.com",
  ]
  tags = local.tags
}

Answer 1

SNS needs to be set-up in conjunction with the Lambda-function and S3-uploads like so (in terraform, excluding KMS for this example):

resource "aws_lambda_permission" "start_from_upload_sns_topic" {
  statement_id  = "AllowExecutionFromSNStopic"
  action        = "lambda:InvokeFunction"
  function_name = module.start_from_upload_handler.arn
  principal     = "sns.amazonaws.com"
  source_arn    = var.upload_notification_topic_arn
}

resource "aws_s3_bucket_notification" "start_from_upload_handler" {
  for_each = var.input_bucket_name_map

  bucket = each.value
  topic {
    topic_arn = var.upload_notification_topic_arn
    events    = ["s3:ObjectCreated:*"]
  }
}

resource "aws_sns_topic_subscription" "start_from_upload_sns_topic" {
  topic_arn = var.upload_notification_topic_arn
  protocol  = "lambda"
  endpoint  = module.start_from_upload_handler.arn
}

The JSON-object the lambda-function receives from the SNS-topic looks like so:

{
    "Records": [
        {
            "EventSource": "aws:sns",
            "EventVersion": "1.0",
            "EventSubscriptionArn": "arn:aws:sns:eu-central-1:...",
            "Sns": {
                "Type": "Notification",
                "MessageId": "...",
                "TopicArn": "arn:aws:sns:eu-central-1:....",
                "Subject": "Amazon S3 Notification",
                "Message": "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"eu-central-1\",\"eventTime\":\"2021-10-27T15:29:38.959Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"AWS:...\"},\"requestParameters\":{\"sourceIPAddress\":\"....\"},\"responseElements\":{\"x-amz-request-id\":\"..\",\"x-amz-id-2\":\"...\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"tf-s3-topic-...\",\"bucket\":{\"name\":\"test-bucket-name\",\"ownerIdentity\":{\"principalId\":\"....\"},\"arn\":\"arn:aws:s3:::test-bucket-name\"},\"object\":{\"key\":\"test_file.json\",\"size\":189,\"eTag\":\"....\",\"versionId\":\"...\",\"sequencer\":\"...\"}}}]}",
                "Timestamp": "2021-10-27T15:29:40.086Z",
                "SignatureVersion": "1",
                "Signature": "...",
                "SigningCertUrl": "https://sns.eu-central-1.amazonaws.com/SimpleNotificationService...",
                "UnsubscribeUrl": "https://sns.eu-central-1.amazonaws.com....",
                "MessageAttributes": {}
            }
        }
    ]
}

We're interested in the "Message" - body of the incoming JSON-object, and this finally looks indeed like @Ermiya Eskandary mentioned in his answer pointing to the S3-notification-JSON-event-structure:

{
    'Records': [
        {
        's3': {
            'bucket': {
                'arn': 'arn:aws:s3:...',
                'name': 'bucket-name',
                },
            'object': {
                'key': 'upload_file_name.json',
                },
            },
        }
    ]
}

The take-away here is that one needs to bear in mind that the incoming JSON emitted by SNS has several top-layer dictionary keywords, which need to be "stripped-off" or "dug-through" in order to get to the actual S3-upload-event in the SNS-Message-body, which comes as a string-JSON-format which needs to be loaded into a proper dictionary-object.

Moreover, it is paramount to subscribe the lambda-function to the SNS-topic and allow the SNS-topic in turn to invoke said lambda-function.