I am using InfiniBand/RDMA for this. My client machine is sending a block to the server (via RDMA Send), and I want the client to get back a special 8 byte acknowledgement. I know the server could send it along separately, but that would duplicate work since my understanding for RDMA Send, the NIC is already sending some form of ACK via the completion queue. However, I was looking at InfiniBand docs, the completion queue doesn't seem to have an immediate along with it, just an id (which is itself less than 8 bytes, so there aren't any funny tricks I could play). I was wondering if I could get suggestions on how to do this?
Thank you!
I'm not sure I fully understand the question, but on the client side, a completion queue entry will be generated when the send work request is fully executed. "Fully executed" means that the RDMA-level ACK has been received from the server in your case. The work request ID in that completion entry is indeed 64 bits, and it is common to use it for a pointer to an auxiliary structure holding whatever info you want for the work request.
If you want the server to actively send back some data that it chooses after it receives the send from the client, then I don't think there's any way around having the server post a work request to do that.