How yo APPEND a AWS S3 bucket Notification Configuration

Question

Trying to append a new notification to a bucket. Couldn't find any example in the internet.

I need to have ObjectCreated events sent to SQS. I need to decide which queue the event is sent by the prefix. Thus, each notification would have a diferent queue and prefix over the same bucket.

The problem is that I cannot append new notification. I just override the previous notification configured in the bucket.

This is the code I have so far:

    bucket_notifications_configuration = {
        'QueueConfigurations': [{
            'Events': ['s3:ObjectCreated:*'],
            'Id': f"Notif_{queue_name}",
            'QueueArn': queue.attributes['QueueArn'] ,
            "Filter": {
                "Key": {
                    "FilterRules": [
                        {
                        "Name": "suffix",
                        "Value": f"{prefix}"
                        }
                    ]
                }
            }               
        }]
    }
    qpolicy = {
        "Version": "2012-10-17",
        "Id": f"{queue_arn}/SQSDefaultPolicy",
        "Statement": [{
            "Sid": f"allow bucket {bucket} to notify queue {queue_name}",
            "Effect": "Allow",
            "Principal": {"AWS": "*"},
            "Action": "SQS:SendMessage",
            "Resource": queue_arn,
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": f"arn:aws:s3:*:*:{bucket}"
                }
            }
        }]
    }

    queue_attrs = queue.attributes
    queue_attrs = {"Policy": json.dumps(qpolicy), }


    queue.set_attributes(Attributes=queue_attrs)
    logger.debug(f"queue ready with attributes: {queue.attributes}")        

    previous_notif = client.get_bucket_notification_configuration(Bucket=bucket)
    previous_notif.pop('ResponseMetadata')

    try:
        print("apendado")
        previous_notif['QueueConfigurations'].append(bucket_notifications_configuration['QueueConfigurations'][0])            
        print(f"apendado {previous_notif} ")
    except KeyError:
        previous_notif['QueueConfigurations'] = bucket_notifications_configuration['QueueConfigurations'][0]
        print("cread")

    client.put_bucket_notification_configuration(
        Bucket=bucket,
        NotificationConfiguration=bucket_notifications_configuration)

I make sure the notification id is diferent from any other, also I make sure the prefix is difefent.

This code overrides previous notification with the new one, instead of appending the new one.

Answer 1

I managed to get it working with the code below.

It takes the existing configurations, adds a new configuration, then saves it back to the bucket. The code assumes that there is an existing configuration.

import boto3

s3_client = boto3.client('s3', region_name = 'ap-southeast-2')

queue_name = 'queue2'
queue_arn = 'arn:aws:sqs:ap-southeast-2:123456789012:queue2'
bucket = 'my-bucket'
prefix = 'folder2/'

# Get the current notification configurations
response = s3_client.get_bucket_notification_configuration(Bucket=bucket)
configurations = response['QueueConfigurations']

# New configuration to add
new_configuration = { 
        'Id': f"Notif_{queue_name}",
        'QueueArn': queue_arn,
        'Events': [
            's3:ObjectCreated:*',
        ],
        'Filter': {
            'Key': {
                'FilterRules': [
                    {
                        'Name': 'prefix',
                        'Value': prefix
                    },
                ]
            }
        }
    }
configurations.append(new_configuration)

# Save combined configurations
response = s3_client.put_bucket_notification_configuration(
    Bucket = bucket,
    NotificationConfiguration = {'QueueConfigurations' : configurations}
)