Search code examples
mysqlamazon-s3amazon-redshiftluigi

How do I write a Luigi workflow for loading data into Redshift from MySQL?

Does Luigi have support for me to execute the entire flow?

The flow would be something like this: Dumping MySQL data to S3, then moving the data to Redshift using the Copy command.

Can I execute the above workflow using Luigi?

Solution

  • Basically you can do anything that you can do with a python script, in Luigi, since you can write that python code in the run() method of Luigi tasks, which means there is no real limits actually.

    See the Luigi docs for tasks, and in particular the run() method, for more information: http://luigi.readthedocs.org/en/stable/tasks.html#task-run

    So, the remaining question is then how to access RedShift and MySQL from python itself, but that is a separate question, for which I think googling around for a good guide or tutorial is the best option.