AWS SQS > EventBridge Pipe > SNS > SQS looses JSON formating

Question

Working in .NET and I have some SQS Queues that are processing Application events around reports.

At one point a report PDF is built and publishes report metadata such as location and ReportId to a Report-Built SQS Queue.

I have an EventBridge Pipe that sources these Report-Built SQS Queue events and pushes the body of the SQS Message to an SNS Topic Report-Ready so that any application can create it's own SQS Queue Subscription to consume the events.

Everything seems fine except when I consume the final SQS Queue Subscriptions the message looses it's JSON Formatting and will not deserialize in my application.

As Published to SQS Report-Built

My EventBridge Pipe extracts the body with the Target Input Transformer <$.body> and publishes it to SNS Topic Report-Ready

At this point I created my SQS Queue to subscribe to the SNS topics for my specific application called Report-Ready-TTL-Monitor. My subscription has the Raw message delivery option Enabled so that I just get my message in the SQS body property instead of it being wrapped in the the SNS message property as well. This is where everything falls apart.

I am expecting my Report-Ready-TTL-Monitor SQS body to be:

{
  "ReportId": "GIUD_STRING",
  "ReportUri": "https://example.com/path/file.pdf"
}

But I am getting a body of:

{ReportId:GIUD_STRING,ReportUri:https://example.com/path/file.pdf}

With all the quotes, whitespace, and line-endings stripped out and then it fails trying to deserialize with JsonConvert.DeserializeObject<ReportReadyNotification>(message.Body)

If I just publish the message directly to my SNS Topic Report-Ready it is received by SQS Queue subscription correctly and everything works.

Any help or guidance about what I am missing here that could be mutating my message body is very much appreciated?

Answer 1

After finding this SO Question and reading the answers by hb2638 kf06925 and reading the AWS EventBridge Pipes Input Transformer documentation again for the thousandth time I finally found a workable solution.

The solution to extracting the body directly is to rebuild the body using the Dynamic JSON feature of the input transformers.

The key here is that if the original input body was JSON it is parsed by the transformer so it can be accessed via JSONPath for transformations. I guess this is why if you de-reference the body as a value without assigning to a Key in a JSON object it strips out the quotes because these extra quotes would result in an invalid string (not sure why they do not just escape the strings again in this case).

As an example if the SQS contains the JSON body:

{
    "StringValue": "Some String Value",
    "DateString": "2023-12-01",
    "TimeStamp": 1701388800,
    "BooleanValue": true,
    "NumericalValue": 5,
    "DecimalValue": 5.5,
    "ArrayValues": [ 1, 2, 3 ],
    "JsonObjects": {
        "IsObject": true,
        "PropertyCount": 2
    }
}

It can be passed through as a JSON Object with the input transformer as follows:

{
  "StringValue": "<$.body.StringValue>",
  "DateString": "<$.body.DateString>",
  "TimeStamp": <$.body.TimeStamp>,
  "BooleanValue": <$.body.BooleanValue>,
  "NumericalValue": <$.body.NumericalValue>,
  "DecimalValue": <$.body.DecimalValue>,
  "ArrayValues": <$.body.ArrayValues>,
  "JsonObjects": <$.body.JsonObjects>
}

VERY IMPORTANT GOTCHAS FOR NULLABLE/OPTIONAL KEYS

ANY KEY of ANY TYPE that is not specifically passed in the JSON Body is assumed to be an empty string "" instead of null.

Any String types you will need to wrap the Body value in quotes inside the transformer because Quotes are not added when values are read from the Body.

So if the string value could be a a well formed string, empty string or null, then it could result in the following possible outputs when transformed:

• Well formed string:"My String Value" -> "StringValue": "My String Value"
• Empty String: "" -> "StringValue": ""
• null: null -> "StringValue": "null"

For Numerical/Boolean/Array/Object types null values work as expected as long as they are either valid values or specifically set to null. This is because the Body value is not wrapped in quotes in the transformer like has to be done for strings. Note that empty Array and Objects are valid and work as expected.