Search code examples
qsubsametime

Can multiple qsub submissions read the same group of files?


I would like to use a bash script and qsub to run 30-40 python programs at once. Each python program reads and searches through the same set of files (~400 total) for a set of sequences. Is there a problem where multiple python programs could be trying to read from the same file? If so, what are the consequences?


Solution

  • There are no problems with multiple jobs reading from the same files that are imposed by Torque. (I think that statement is likely to be true for any resource manager / scheduler.)

    The main issue I can imagine would be your file system's performance and whether or not it can keep up with potentially concurrent accesses to those files.