Run same Oozie job multiple times in parallel, with different values in argument for each job

Question

I want to run a program that converts given input file to another format. When a file is uploaded to input table, oozie scheduler should start the job.

For single file it is working fine; If input table have multiple entries at same time, same number of jobs should start parallely with different inputPath in arguments. Oozie workflow takes "InputPath" and "OutputPath" as argument.

How can I do this?

Answer 1

You want to trigger jobs based on events. Oozie was not designed for that, it just runs scheduled jobs based on a time table (w/ optionally a deferred start until arrival of specific files).

Consider other options, e.g. AirFlow or Luigi.