Search code examples
nodessubnetinfiniband

testing 3 nodes at once

I've been using the Infiniband commands (such as ib_send_bw, ibv_rc_pingpong, etc) to test latency and bandwidth between nodes (through the infiniband switch). However, I want to test more things. Is it possible to test latency and bandwith when two nodes send packets to a third node at the same time, or one node sends packets to two other nodes? If so, how?

Thanks

Solution

  • Unfortunately, the libibverbs benchmarks are not sufficient for your case (for example, multiple nodes talking to one node). You need to write benchmarks of your own or look at existing code.

    Take a look at the OSU benchmarks: http://mvapich.cse.ohio-state.edu/benchmarks/. I think some of them can measure many-to-many communication performance.

    Here is a system (that I wrote) in which multiple client machines communicate with one server machine: https://github.com/anujkaliaiitd/HERD. In the default configuration, 36 client processes communicate with 6 server processes. The clients issue requests to the servers using RDMA writes over UC transport. The servers reply using SEND messages over UD transport.

    I cannot guarantee that it will work for you. However, if you get bugs, I can offer some help.