I am running an app on heroku and I wanted to perform a map/reduce style operation, where I can fire jobs to workers and then once the results are done, fire the reduction jobs.
This would mean I need to call out some (reduce) jobs as being dependent on others (map).
Can I achieve this using heroku?
I saw this fork, but is not actively under development. https://github.com/mceachen/delayed_job
For example, a distributed merge sort -- with sort jobs and merge jobs and merge jobs depend on the data from the sort jobs.
DelayedJob has been forked around quite a bit, and the repos can be misleading. It's still in active use and development via this repo:
https://github.com/collectiveidea/delayed_job
I do believe that DelayedJob can support what you're asking for. You can start heroku workers to run rake jobs:work, which will start a worker munching on job entries in the DelayedJob queue.
a) If it can be ensured that your sort jobs and your merge jobs enter the DelayedJob queue in a required order, then your dequeuing worker can interleave processing of sort and merge jobs, given that the merge jobs have a dependency on a sort job having just occured.
b) Could the sort job and merge job be packaged into one 'job', wherein a worker dequeues this pair of operations as a whole, and processes them both? This would be a better solution if you want to scale up your number of heroku workers and have multiple, concurrent DelayedJob workers chomping on your DJ queue simultaneously.