How to send events from Azure Event Grid to AWS EventBridge

Question

We are working on a project which needs to route events created in Azure back to AWS infrastructure.

High level overview:

• When a new object is added to our Azure storage account, it generates an event that contains information such as the object name and size.
• We have a Lambda function on AWS that processes these events and updates our DynamoDB.
• Our goal is to transfer these events generated in Azure to AWS to invoke the Lambda.

We have explored multiple approaches, including directly invoking a Lambda using Azure functions. However, we are also considering the possibility of using Azure EventGrid, which is a special event broker, along with AWS EventBridge.

With this approach, we can utilize EventGrid at the source (Azure) and EventBridge at the destination (AWS). This will enable EventBridge to invoke different Lambda functions and SQS as per our use case.

We would appreciate any help in understanding if this integration is supported by both AWS and Azure, if it is feasible, and if there are any other alternatives we can explore.

Answer 1

We have adopted below approach to send events from Azure Event Grid back to AWS infrastructure

Summary of steps:

1. Create Azure storage and containers
2. Create Event Grid topic to receive storage events
3. Create Subscription with custom webhook
4. On AWS side create API Gateway webhook where Azure Event Grid can post the events
5. Create API Gateway integration with SQS to route events to desired SQS

Resources Required (for event transfer):

• Azure Event Grid Topic
• Azure Event Grid Subscription
• AWS API Gateway Webhook

Detailed Steps: (Provided manual steps)

Step 1: Create IAM Role (AWS Side) Create new IAM role which allows API gateway permission to put messages to SQS or EventBridge event bus Check below IAM policy for permissions details, also add "apigateway.amazonaws.com" in the Trust relationship.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Resource": [
                "{ARN of SQS}"
            ],
            "Action": [
                "sqs:SendMessage"
            ]
        }
    ]
}

Step 2: Create API Gateway Webhook (AWS Side)

Expected Output: After following below steps is an API which puts events to SQS or EventBridge

1. Create a new REST API
2. Create a new "Post" method resource and Stage
3. In the "Integration Request" Do following
4. Select Integration type: AWS Service
5. Select Region e.g. "us-west-2"
6. Select AWS Service as "Simple Queue Service (SQS)" Or "CloudWatch Event" (for EventBridge integration)
7. Select HTTP Method as "POST"
8. For "Path override" Provide {AWS Account ID/SQS Name}
9. Provide ARN of IAM role created in Step 1
10. Select "Do not add caller credentials to cache key" for Credentials cache For Content Handling select "Passthrough"
11. Select "Use default timeout" and For HTTP Headers, for "Name" parameter provide 'application/x-www-form-urlencoded'
12. For Mapping Templates, For "Request body passthrough" select "When there are no templates", For "Content-Type" provide application/json and For template body, provide "Action=SendMessage&MessageBody=$input.body"

Step 3: Create Event Grid Subscription (Azure Side)

Expected Output: After following below steps we will have Event Grid subscription which will route file created events to AWS Webhook

1. Go to Event Subscription and create new subscription
2. Provide Name and For Event Schema select "Event Grid Schema"
3. For Topic Types select "Storage Accounts (Blob & GPv2)"
4. Provide Subscription name
5. Provide Resource Group Name
6. For Resource, Select the storage account for which we need to capture events
7. Select Filter to Event Types, here we can select "Capture File Created"
8. For "Endpoint Type" Select "Web Hook"
9. For "Endpoint", Provide the API gateway Invoke URL as subscriber endpoint

Step 4: Endpoint (Webhook) handshake with Event Grid

Expected Output: The statues of event subscription as "Success"

1. Once we create Event Grid subscription, it will try to validate the integrated webhook
2. For custom webhooks we need to perform "Asynchronous handshake" where the Webhook needs to respond back with HTTP status code 200 Ok, status
3. Also we need to retrieve the "validationUrl" from the API request and perform "GET" on the URL received to complete the handshake
4. The event subscription URL will be valid for 5 minutes and the subscription status will remain "AwaitingManualAction"
5. If there's a GET on the validation URL within 5 minutes, the validation handshake is considered to be successful.