I've been having a very confusing discussion with a colleague about the use of Network Direct in Windows to provide RDMA, and Linux where we have a working implementation using the verbs api and rdmacm.
Said collegue developed his Network Direct implementation after he couldn't get the verbs one to work. I did the rdmacm implementation under Linux. We both got them talking linux=>linux and windows=>windows but never got around to testing Linux=>windows or windows=>linux. During the development there was some hint of it possibly working, they appeared to start talking but of course things were still in development.
The project was put on hold for months and the framework that it was built into has had significant redesign. I'm left with pulling what was working from the Windows side into the new framework and I'm having a hard time getting it working or seeing any part of it connect from Linux.
My understanding is that at the lower layer they will both talk verbs. But I'm now not so sure with colleague saying things like "Remember this is not verbs", just to add to the confusion.
So the question is. will an application that uses rdmacm be able to establish a connection to an app that uses Windows NetworkDirect interface or are the two completely different and incompatible?
According to Mellanox support, Network Direct should be compatible with InfiniBand verbs. We've had good luck doing RDMA transfers using rdmacm Linux-Linux and Windows-Linux. We used the windows libraries available at open fabrics alliance (look for ofed_sdk). It was not easy to figure out what libraries to get in order to do the Windows part.
However, Microsoft does not want people to use the ibverbs approach on MS-Windows any more. If you are having trouble getting ND on Windows to talk to Linux, are you sure that you are putting the values in network order before shipping them over?