Retry Logic when some things fails in step function python

Question

I have a scenario where I need to re-run the function, how can I do that.

def test_cpbucket():

    res = s3.copy_object(
       Bucket = 'bucket-name',
       CopySource: 'bucket-name/hello/21-01-2020-12-23-00/testing.ddl',
       Key: 'hello/21-01-2020-12-23-00/testing.ddl',
       ACL: 'public-read' 
    )

in the above example, am copying objects to the same location so that it can trigger my lambda function.

I need to re-run this function of copy s3 when ever lambda function fails. How can i do that ?

Answer 1

you can try something similar of below step function. step function definition

Steps :

1. start at s3-copy-task - which executes the lambda.
2. s3-copy-task-choice - which makes a choice - rerun, success, failure. It makes the decision based on LastEvaluatedKey and exception variables returned from the lambda function in step 1.
3. s3-copy-task-fail - in case the whole process is failed some post processing or clean up task can be performed here.
4. s3-copy-task-success - in case the whole process is success some post processing or clean up task can be performed here.

Step function code as below,

{
  "StartAt": "s3-copy-task",
  "States": {
    "s3-copy-task": {
      "Next": "s3-copy-task-choice",
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:eu-west-1:474496007096:function:ph_offer-synchronization-dev-query-dynamodb-fn",
        "Payload.$": "$"
      }
    },
    "s3-copy-task-choice": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.Payload.exception",
          "StringEquals": "true",
          "Next": "s3-copy-task-fail"
        },
        {
          "Variable": "$.Payload.LastEvaluatedKey",
          "StringEquals": "EMPTY",
          "Next": "s3-copy-task-success"
        }
      ],
      "Default": "s3-copy-task"
    },
    "s3-copy-task-fail": {
      "Type": "Fail",
      "Cause": "Error during lambda processing"
    },
    "s3-copy-task-success": {
      "Type": "Succeed"
    }
  }
}