I have an MPI program that uses a master process and multiple worker processes. I want to have the master process running on a single compute node alone, while the worker processes run on another node. The worker processes should be assigned by socket (for example as it is done with the --map-by-socket option). Is there any option to assign the master process and the working processes to different nodes or to assign it manually, by consulting the rank maybe?
Thanks
Assignment of ranks to hosts simultaneously with binding is possible via the use of rankfiles. In your case, assuming that each node has two 4-core CPUs, something like this should do it (for Open MPI 1.7 and newer):
rank 0=host1 slots=0-7
rank 1=host2 slots=0:0-3
rank 2=host2 slots=1:0-3
For older versions, instead of slots=0:0-3 and slots=1:0-3 one should use slots=0-3 and slots=4-7 respectively (assuming that cores are numbered linearly which might not be the case). Then the rankfile is supplied to mpiexec via the --rankfile option. It supersedes the hostfile.
Another option would be to do an MIMD launch. In that case one could split the MPI job into several parts and provide different distribution and binding arguments for each part:
mpiexec -H host1 -n 1 --bind-to none ./program : \
-H host2 -n 2 --bind-to socket --map-by socket ./program