AWS AppFlow Salesforce to Lambda incomplete events

Question

I have an AWS AppFlow flow setup with the intentions of receiving Salesforce change events and processing on a Redshift table in order to keep the Redshift table current as changes happen throughout the day in Salesforce.

Using just AppFlow and mapping source to target fields in Redshift doesn't seem to do what I want as it appears to just append changes to the target Redshift table - no updates, deletes, etc.

I figured the way around this would be to change the target of AppFlow to Amazon EventBridge, setup a rule to listen for changes, target a Lambda function, and perform the Redshift operations on the appropriate tables based on the event details.

However, after making these changes and testing in Lambda, I've run into an issue with the events. According to Salesforce a Change Event Message should look like this:

{
  "data": {
    "schema": "<schema_ID>", 
    "payload": {
      "ChangeEventHeader": {
         "entityName" : "...",
         "recordIds" : "...",
         "changeType" : "...",
         "changedFields": [...],
         "changeOrigin" : "...",
         "transactionKey" : "...",
         "sequenceNumber" : "...",
         "commitTimestamp" : "...",
         "commitUser" : "...",
         "commitNumber" : "..."
      }, 
     "field1":"...",
     "field2":"...",
     . . .
    }, 
    "event": {
      "replayId": <replayID>
    }
  }, 
  "channel": "/data/<channel>"
}

When I go in to Salesforce and update a record, the event that gets sent to Lambda looks like this:

{
    'account': '111111111111',
    'detail': {'my_custom_field__c': 'my updated text'},
    'detail-type': 'AccountChangeEvent',
    'id': '7b205b55-879a-1wt2-br8d-227da614b205',
    'region': 'my-region',
    'resources': [],
    'source': 'aws.partner/appflow/salesforce.com/111111111111/aws.partner/appflow/salesforce.com/222222222222',
    'time': '2021-07-09T15:29:21Z',
    'version': '0',
}

The only Salesforce related information being in the detail key which are just the fields without any of the ChangeEventHeader information. Just the fields by themselves aren't useful, I need the IDs to know which row to update in Redshift.

Are the fields being filtered out somewhere? Any ideas on where to look to obtain the whole event payload? I'm expecting the whole payload, not just a partial.

-update- I changed the target of my flow to S3 and inspect the JSON output and it is exactly as I need it to be. So the issue seems to be when events are received from EventBridge:

{
    'ChangeEventHeader': {
        'changeOrigin': 'com/salesforce/api/soap/52.0;client=SfdcInternalAPI/',
        'changeType': 'UPDATE',
        'changedFields': ['LastModifiedDate', 'my_custom_field__c'],
        'commitNumber': 34143472625437,
        'commitTimestamp': 1655870833010,
        'commitUser': '0044r00000a5VaPcdv',
        'entityName': 'Account',
        'recordIds': ['0034t00003bOPtuSJs'],
        'sequenceNumber': 1,
        'transactionKey': '02030d57-6dd7-82f8-482r-7pou3bb58769',
    },
    'my_custom_field__c': 'my updated text',
    'LastModifiedDate': '2021-07-09T17:02:13.000Z',
}

Answer 1

In the Source to destination field mapping section in AppFlow the keys for the event are listed here. I checked ChangeEventHeader to include in the flow and I now have the full event passed to EventBridge and to Lambda.