We had a micro service approach for one of our systems using Kafka as an event bus. We had some latency problems and experimented with replacing Kafka topics with a bunch of Chronicle queues. When running locally on a developer machine the results were amazing, one of our most expensive work flows was processing ten to thirty times faster.
Given the initial good results we decided to take the experiment further and deploy our proof on concept in AWS which is where our system runs. Our micro services run in docker containers across a bunch of EC2s.
We created an EFS volume and mounted it on each docker container. We verified the volume was accessible from each micro service and the right read write permissions were granted.
Now the problem: MS1 receives a message (API call) does some processing and emits an event in a chronicle queue. We can see on the EFS file system the chronicle queue file is touched. MS2 is supposed to consume that event and do some further processing. This is not happening. Eventually restarting MS2 would trigger the message processing but this is not always the case. Easy to imagine the disappointment.
The question: Is our EFS approach wrong? If yes what would be the way to go?
Thank you in advance for your inputs.
Chronicle Queue cannot work on a Network File System like EFS, as discussed in this previous question and also documented here: https://github.com/OpenHFT/Chronicle-Queue/#usage
To communicate between hosts you need Chronicle Queue Enterprise which supports TCP/IP replication.
Please note also doco for running with docker