How to send AWS datapipeline failure messages somewhere other than SNS?

Question

I want to collect failure logs for AWS datapipeline actions. The standard failure handling provided by AWS when datapipeline actions (steps) fail is to send a notification to a SNS topic. I want to store these failures in a log store, instead. (Ideally, in S3).

Is there any way to configure my datapipeline to log these failure notifications instead of pushing a notification to SNS? I want to send datapipeline failure logs to S3.

According to the documentation, it appears that SNS is the only supported destination for failure notifications.

I am hopeful that there is a way, because it is a very common pattern in AWS to be able to push logs to S3 buckets.

I wonder, would I be able to find failure logs in the pipelineLogUri S3 bucket that is configured for each datapipeline step? (If I can filter these granular logs for the fatal errors, that would be acceptable to me)

Answer 1

I decided to subscribe a lambda function to the SNS notifications and push them to S3 from there. Not ideal (because it adds additional complexity to our system), but it works.