Have previously worked with lambda orchestration using AWS step function. This has been working very well. Setting the result_path of each lambda will pass along arguments to subsequent lambda.
However, I now need to run a fargate task and then pass along arguments from that fargate task to subsequent lambdas. I have created a python script that acts as the entrypoint in the container definition. Obviously in a lambda function the handler(event, context) acts as the entrypoint and by defining a return {"return_object": "hello_world"} its easy to pass a long a argument to the next state of the state machine.
In my case though, I have task definition with a container definition created from this Dockerfile:
FROM python:3.7-slim
COPY my_script.py /my_script.py
RUN ln -s /python/my_script.py /usr/bin/my_script && \
chmod +x /python/my_script.py
ENTRYPOINT ["my_script"]
Hence, I am able to invoke the state machine and it will execute my_script as intended. But how do I get the output from this python script and pass it along to another state in the state machine?
I have found some documentation on how to pass along inputs, but no example of passing along outputs.
To get output from an ECS/Fargate task, I think you have to use the Task Token Integration instead of Run Job (Sync) which is usually recommended for Fargate tasks. You can pass the token as a container override ("TASK_TOKEN": "$$.Task.Token"). Then inside your image you need some logic like this:
client = boto3.client('stepfunctions')
client.send_task_success(
taskToken=os.environ["TASK_TOKEN"],
output=output
)
to pass it back.