I'm creating a proof of concept implementation in which I need to send files (sometimes large ones, up to 400Mb) embedded in messages using AMQP 1.0. I decided to use Apache's qpid proton client lib and qpid C++ broker.
The test setup has 3 parts: A sender program, the broker, and a receiver program. I'm using SSL for transport encryption and SASL for authentication. All parts run on the same box, connected through the loopback interface.
Everything works fine, except that the performance when sending files from the sender to the broker is really bad. Example: I sent a 140Mb message, and it took 90 seconds from the sender to the broker. The same message took only 2 seconds to deliver from the broker to the receiver.
The queue is not persistent, so no I/O bottlenecks.
I know that AMQP is not designed to send large messages, but seeing that it can actually deliver it fast enough on the receiving end, I figured it's capable of meeting my requirements, and I might be doing something wrong on the sending side.
So, the questions are:
I'm going crazy about this issue, so any help or idea that can point me to a solution is very welcomed. Thanks in advance!!!
AMQP can be used to send large message but how that gets done will depend a lot on the client and server implementations. If sending a large message it can be a good idea to lower the max frame size value to something smaller than the default which will usually be something like 2 GB frames (I don't know what the Qpid C++ client or broker do here) as that can allow the IO layer to send chunks of the delivery as opposed to waiting for the full thing to be written by the client application.
The client API can also be a factor as it needs a means of allowing for writing of partial bits of a larger message without waiting for the full value to be fed into the client sender (again I don't know if that's possible in the C++ client). The proton-c engine implementation combined with the API that I see in the docs would suggest to me that the client needs the full message loaded into memory and then inside proton will create another copy of the message for the framing layer and possible more copying depending on how framing ends up being done (one frame or many). I'd guess this has a lot to do with the overall sending time.
I'd advise against using the Qpid broker as it isn't actively maintained so you won't likely see any fixes or updates if you run into issues. An actively maintained broker such as ActiveMQ Artemis might be a better choice for a broker that can deal with large messages.
There is a Java client implementation at Qpid that does have an API purpose built for sending large messages which you can find here. There is an example of sending a file here.