How to handle missing values in JSON when processing AWS Step Functions

I have a Step Function that takes a JSON payload. The handler accepts a Map<String, Object> and returns the same thing.

The issue I have run into is keys are lost when serializing the map back to JSON which causes me exceptions at the next step.

I have the following which transforms the input, into a common structure for the lambda:

"SearchForPersonCustomer": {
     "Type": "Task",
     "Resource": "arn:aws:states:::lambda:invoke",
     "Parameters": {
     "FunctionName": "arn:aws:lambda:us-west-2:xxxxx:function:searchForCustomer:$LATEST",
     "Payload": {
           "businessSearchInd": false,
           "taxIdentifier.$": "$.sSN",
           "firstName.$": "$.firstName",
           "middleInitial.$": "$.middleInitial",
           "lastName.$": "$.lastName",
           "birthDate.$": "$.birthDate"
         }
     }

The issue is that while the field middleInitial ("middleInitial":null) is populated in the JSON, the prior steps output no longer serializes it, so when it attempts to access it, the step function fails as follows:

The JSONPath '$.middleInitial' specified for the field 'middleInitial.$' could not be found in the input

Is there a way to handle this edge case, so that if the missing field, in this case middleInitial is just assumed to be null as it translates to the payload?

Thanks

Solution

You can do one of two things: If this is coming from an API then you can set up a Method or Integration Request Transformation - this will enable you to add a default value if its missing. See this documentation.

Alternatively, if you're not dealing with an API then your best bet is to add a translation layer lambda as your first step - which accepts any event and checks for the required fields - it can refuse it if you can't default or it can default those values for you, depending on your requirements.

Step Functions itself is a "dumb" system - it can do some very basic choice and repeating logic but it cant make assumptions or parse or set things if they are missing. Thats what Lambdas are for - to handle that sort of stuff for your step function. the Step Function should just be a framework for transfering data to the right lambda/resource/service