I am using a Apache nifi for one of my clickstream projects to do some ETL.
I am getting traffic around 300 messages per second currently with the following infra:
The entire cluster UI has become extremely slow with the following issues
I have the following questions on the setup
A couple of suggestions
Concurrent Tasks
. This is more of trial and error or testing and fine tuning approach that you'd have to master. One suggestion is, if you set too many Concurrent Tasks
for a CPU intensive processor, it will have serious impact on the nodes.Some best practices and handy guides: