Luigi : The task does not fail even if in the run() method ,i execute a file which does not exist

Iam new to luigi and exploring its possibilities. I encountered a problem wherein I defined the task with (requires ,run and output method). In run(), I'm executing the contents of a file.

However , if the file do not exist , the task does not fail . Is there something I'm missing ?

import luigi
import logging
import time
import sys, os

logging.basicConfig(filename='Execution.log',level=logging.DEBUG)
date = time.strftime("%Y%m%d")

class CreateTable(luigi.Task):
    def run(self):
        os.system('./createsample.hql')
#       with self.output().open('w') as f:
#               f.write('Completed')

    def output(self):
        return luigi.LocalTarget('/tmp/CreateTable_Success_%s.csv' % date)

Output :

INFO: [pid 15553] Worker Worker(salt=747259359, workers=1, host=host-.com, username=root, pid=15553) running CreateTable() sh: ./createsample.hql: No such file or directory INFO: [pid 15553] Worker Worker(salt=747259359, workers=1, host=host-.com, username=root, pid=15553) done CreateTable() DEBUG: 1 running tasks, waiting for next task to finish INFO: Informed scheduler that task CreateTable__99914b932b has status DONE

Solution

Technically your code works and the Python part of your job ran successfully. The problem is that you are doing a system call that fails because the file does not exist. What you need to do here is to check the return code of the system call. Return code 0 means it ran successfully. Any other outcome will yield a non-zero return code:

rc = os.system('./createsample.hql')
if rc:
    raise Exception("something went wrong")

You might want to use the subprocess module for system calls to have more flexibility (and complexity): https://docs.python.org/2/library/subprocess.html