How to filter input data and pass to AWS step function?

Question

One of the step in the step function outputs a list of objects. I want to filter the list based on Status and store the objects in separate files in s3 (e.g. file with list of objects with success vs. failed). Is there a way to filter the data in the step function? Or do I have to create a lambda to filter the data and then, create another lambda to write each filter list of objects to s3?

Input:

[{
 Name: "Test 1",
 Status: "Success",
 Data: {...}
},
{
 Name: "Test 2",
 Status: "Success",
 Data: {...}
},
{
 Name: "Test 3",
 Status: "Failed",
 Data: {...}
}]

Answer 1

Assume you are going to process the list of objects split into 2 piles at the same time.

Filtering with JSONPath Syntax

You can use a filter expression with JSONPath syntax for your case. It indeed has some limitations currently yet in your scenario, this filtering expression, $[?(@.Status==Success)], shall work for you.

• $: Refers to the root of the JSON document.
• [ ]: Square brackets indicate that we are working with an array.
• ?(): A filter expression, denoted by a question mark and parentheses. It is used to apply a condition to the elements within the array.
• @: Refers to the current element in the array.
• .Status: Accesses the Status property of the current element.
• ==Success: This is the condition that must be true for the element to be included in the result. It checks if the Status property of the current element is equal to the string 'Success'.

Data Flow Simulator Results

This is the result of checking the filtering expressions using the data flow simulator available in AWS Step Functions.

Minimal ASL Reference

{
  "Comment": "A description of my state machine",
  "StartAt": "Some compute progress",
  "States": {
    "Some compute progress": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "OutputPath": "$.Payload",
      "Parameters": {
        "Payload.$": "$",
        "FunctionName": "arn:aws:lambda:ap-northeast-1:123456789012:function:some-compute-progress:$LATEST"
      },
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException",
            "Lambda.TooManyRequestsException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Next": "Parallel"
    },
    "Parallel": {
      "Type": "Parallel",
      "End": true,
      "Branches": [
        {
          "StartAt": "Process success",
          "States": {
            "Process success": {
              "Type": "Task",
              "Resource": "arn:aws:states:::lambda:invoke",
              "OutputPath": "$.Payload",
              "Parameters": {
                "Payload.$": "$",
                "FunctionName": "arn:aws:lambda:ap-northeast-1:123456789012:function:process-objects-expert:$LATEST"
              },
              "Retry": [
                {
                  "ErrorEquals": [
                    "Lambda.ServiceException",
                    "Lambda.AWSLambdaException",
                    "Lambda.SdkClientException",
                    "Lambda.TooManyRequestsException"
                  ],
                  "IntervalSeconds": 2,
                  "MaxAttempts": 6,
                  "BackoffRate": 2
                }
              ],
              "End": true,
              "InputPath": "$[?(@.Status==Success)]"
            }
          }
        },
        {
          "StartAt": "Process failed",
          "States": {
            "Process failed": {
              "Type": "Task",
              "Resource": "arn:aws:states:::lambda:invoke",
              "OutputPath": "$.Payload",
              "Parameters": {
                "Payload.$": "$",
                "FunctionName": "arn:aws:lambda:ap-northeast-1:123456789012:function:process-objects-expert:$LATEST"
              },
              "Retry": [
                {
                  "ErrorEquals": [
                    "Lambda.ServiceException",
                    "Lambda.AWSLambdaException",
                    "Lambda.SdkClientException",
                    "Lambda.TooManyRequestsException"
                  ],
                  "IntervalSeconds": 2,
                  "MaxAttempts": 6,
                  "BackoffRate": 2
                }
              ],
              "End": true,
              "InputPath": "$[?(@.Status==Failed)]"
            }
          }
        }
      ]
    }
  }
}

References

1. Beswick, J. (2021) Using JSONPath effectively in AWS Step Functions, AWS Compute Blog. © 2023, Amazon Web Services, Inc. or its affiliates. Available at: https://aws.amazon.com/blogs/compute/using-jsonpath-effectively-in-aws-step-functions/ (Accessed: April 7, 2023).
2. Liszi (no date) Step functions Filter support for jsonpath - feature request, AWS re:Post. Available at: https://repost.aws/questions/QUkLTFPfR7TwqEo4JD_Lvh7w/step-functions-filter-support-for-jsonpath-feature-request (Accessed: April 7, 2023).