I'm wondering if there is a good way to transfer heavy objects (for example, arrays of >1mln strings) effectively via Rabbit queue.
What I tried was splitting the array on chunks by 1000, 10000, and 100000. The fastest I got was 21sec with 10000 chunk size.
The solution I came up with was something similar to streaming. There were 2 services, one sent to another a request object containing the temporal queue name (which was created right before sending the request). The latter gets the request and starts sending the array by chunks, and when it's over, it sends a kind of END-OF-STREAM sign.
Maybe there's some elegant solution for that, like some feature or library or whatever. The only condition is that the heavy object should be passed by using a message queue.
Have anyone encountered such a task?
Ok, I haven't gotten any response here so just tell you some more about my solution. What I did was, I just create and declare a temporal queue in the first service and the second service has already had the queue to listen to requests. So service #1 sends a request to the queue, and the request DTO contains the temporal queue name.
After sending, service #1 starts listening to the temporal queue like that:
while (condition) {
Object obj = amqpTemplate.receiveAndConvert(queueToListen, DEFAULT_TIMEOUT);
}
DEFAULT_TIMEOUT is how long service #1 is going to listen to the temporal queue. The loop breaker could be, for instance, sent from service #2 as a special object that notifies that that was the end of the stream.
To achieve more performance, you can also simplify your DTOs that are being passed through the temporal queue. Namely, if they all have some common fields, they could be sent once (in the stream header or whatever) and excluded the chunk DTO. Additionally, the JSON property names could also be shortened (e.g. value => v) because essentially we still pass bytes of information, and the objects are serialized.