I'm developing a project where I need to reduce a video resolution from 1080p to 720p. That part is pretty straight forward with the command "ffmpeg -I inputVideo.mp4 -r 60 -s hd720 outputVideo.mp4".
Now, the idea is using HTCondor, so every Resource receives one part of the video (Scatter), reduces it quality, and returns it, so in the end all parts can be put together to make the full video 720p (Gather). The issue is, every node returns its own segment, so how can I make that the return is the total junction video. I thought of using a .dag file, so after the execution, it would run a script that would merge them, but, for what I understand from the documentation that is run on each node and not in a place where all segments are, which is the client, I guess.
So I would like to ask, what do you suggest so I can merge the output video files.
I think you are on the right track with dagman. Start from a diamond dag like in the docs:
But maybe you want more than 2 nodes to do the processing -- the scattering. As each of these HTCondor jobs finishes, the output will be sent back to the submitting machine. You can either send all of these out to a HTCondor worker node for the final assembly, or, if the assembly is not a cpu-heavy activity, just run it on the submit machine where all the intermediate outputs are. In HTCondor,a "local universe" job looks just like a normal job, but runs on the submit machine. Perhaps the final node that depends on all the scatter nodes could be a local universe job.
At a training session for dagman, there was a similar image processing example with the mandlebrot set. This link may give you some good ideas:
https://opensciencegrid.org/user-school-2019/#materials/day4/part4-ex1-simple-dag/