Search code examples
pythonetlluigi

How to make a Luigi task generate an in-memory list as target


I'm trying to write an etl pipeline using luigi. As far as I understand from the documentation a task in luigi can generate a target that can be either some type of file storage or a database. To decrese the processing time I would like to have as an output an in-memory list. Is this possible? Do I have to create a custom target?


Solution

  • I found out I can use a MockFile as a target. A good example:

    http://gouthamanbalaraman.com/blog/building-luigi-task-pipeline.html