I got two questions regarding Slurm srun.
When we run "srun ", how does slurm know that is a MPI executable or not? In case of actually running an MPI executable, it must pass certain information to the MPI environment right? If it is not an executable, srun simply run it on all available resources?
Is there any difference between running "srun " in a sbatch script and runing it in an interactive mode that has the same set of resources allocated? They both just instantiate ntasks processes using the on the allocated resources of the sbatch script or interactive mode right?
Slurm does not actually know. It sets up the environment it is configured to setup, and the MPI library of your program uses it to start IPC. You can start a bash script and inspect this environment yourself. This is usually done via PMI (great name for dyslexic folks) and you can see variables with that name pop up in env. https://www.mcs.anl.gov/papers/P1760.pdf
There has to be some difference, but for all intents and purposes the two behave the same, someone else can get that one.