I was asked a question to achieve scaling a system. The system itself processes customer data and creates filtering on that data and produces some analytical information.
1) The First iteration:
My initial answer was to provide a Kafka cluster solution.
Kafka itself has streaming , load balancing, and fault tolerance capacity. So it provides me to create processing the producing data in many brokers effectively and consume this data in any consumer as I want.
I can also add streaming functionality to filtering data as much as I want.
In this scenario, there is no need to think about the load balancers, because the Kafka handles the load balancing itself.
2) The second iteration: It has been asked to me how do I scale the system if there is huge demand increase. What would be the approach to scale the system?
In this case, when considering the Kafka cluster; broker counts and partitions should be described in the beginning. It is not something expanding itself. So even though Kafka provides lots of flexibility when considering multiple locations and fastly increasing requests my second opinion is using Elastic Load Balancers & automatic binding for data centers.
When the request counts double in the next day. Load balancers route the load other load balancers/ other data centers so in case of necessity the new Kafka clusters automatically connect to the total system.
The main load routing could be done geographically.
The point of this issue load balancers is still looks needed although Kafka is so strong candidate.
My second approach similar to the following schema.
https://i.sstatic.net/kEx1C.jpg
(In the meantime, I came across some interviewers before than this interview and they exactly named load balancer as an "out of date technology" and I have been judged very cruel because I suggested load balancers. )
If you are a Kafka expert & and dealing with scaling of increasing requests in multi-location, I would be glad if you put your comments.
Thanks.
Load-balancing with Kafka would run into problems, as the client itself is going to create multiple connections to Kafka brokers, possibly bypassing your proxies. On startup, clients (producers/consumers) send Metadata requests to the bootstrap.servers figure out how the cluster looks like. This can be observed in detail when you turn on trace/debug log levels in your Java clients.
The mesh-level solution, on the other hand, is going to need protocol support, e.g. something like this is happening in Envoy - https://github.com/envoyproxy/envoy/issues/2852
