As part of a Luigi pipeline we want to notify microservices waiting for the data being computed using a POST request.
Until now we were using the RunAnywayTarget but it is a problem if we fire up Luigi faster than the rate of data change. So my question is,
what is the best pattern to create a task that does something in the pipeline but that doesn't create any piece of data, like doing a POST request to a REST service, send a message to Kafka, etc...?
I know that I could create a task with no output that does the request in the run method, but then how should this NotificationTask be re-run again if for some reason the end service failed during the first run? The dependencies will be there and it won't be run again.
I would go ahead and create an unique output for the task, even if the output is not used in your further processing. It would just be a marker that the task with the particular set of inputs has completed successfully. You could do a simple FileTarget, a PostgresTarget, etc.