Suppose I have an HPC cluster with 27 nodes. I want to perform 100 jobs that would each take 10000 seconds of compute time. If a single node did this entire job serially, that works out to 100 x 10000 seconds, or 11.57 days. My first assumption is that I can simply divide 11.57 by 27 to get a cluster compute time of 0.43 days. Is that a good estimate? I feel like that it could be more complicated and I'm missing something. Perhaps I'm just overthinking... I just need the most accurate assessment possible.
Using Amdahl's law (https://en.m.wikipedia.org/wiki/Amdahl%27s_law), split the total time into the part that must be done serially, and the part that can be done in parallel. Divide the second part by the number of processing nodes you have, add the first part, and you have a good estimate of how long it will take.
In general, if your jobs are largely independent then the only part that must be done serially is starting the jobs and harvesting results, if any, that must be communicated back to the central processor.