Search code examples
apache-kafkarabbitmqmessaging

Message bus best practices to send updates


I would like to understand what are the best practices when using a message bus to update several services.

The basic show case is to send the full object through the bus and everyone listening can immediately get the data and process it.

Other option is to send the minimal necessary data through the bus, 3/4 fields, and then each listening service should request the main application to fetch the required data to process it.

The main application is not waiting for anyone to complete their work, and we are talking about an object with 30 to 40 fields and no more than 10kb, most usually less than 5kb.

Is there any architectural advantage using the second approach?


Solution

  • Design is always about tradeoff between options.

    Option-1: This is preferred and simple to implement and truly decouple from producer. If require compression can be enable to reduce the size, though compression can slightly reduce the overall throughput. This approach will require more disk space and application memory. Ex: Json payload. Your example appears to be more suitable with option-1.

    Option-2: This can be use if purpose of kafka message is to notify with minimal information and additional details could be file or independent object/multimedia urls. One Reference implementation is in AWS S3 event streaming, where S3 publish the message arrival information(ex:someone uploaded file) and S3 url, then consumer can use this information to process. This approach require object storage or multimedia host to allow the consumer to connect and download/process particular event.