Search code examples
pythonluigi

Is it possible in luigi Python to use database row as Output

I am learning luigi and see that mostly the output is file on filesystem.

If that file exists then luigi think that task is done.

In my case instead of writing file i want to update databas erecord with status DONE in postgres.

I want to know that is it possible in luigi

Solution

  • Take a look at Luigi contribs. There are at least 3 packages there you can be interested in: mssqldb, mysqldb, rdbms, sqla and redshift. If you open any of them, look for luigi.Target subclasses.

    By instance, in luigi.contribs.mysqldb you have MySqlTarget. It gives you a functionality like the one you talk about above: the task checks that the "marker table" exists (if not, this task will create it), and then makes a select to check if the task has been already executed successfully. If not, it inserts a "marker" for the task saying it's DONE.

    If you already have a "marker table" you can subclass this task class or create a similar one by copying its code and customizing the parts you need or don't need.