While performing the load testing of the cadence cluster, we are seeing extremely high number of threads ( > 4000) and it constantly remains like this consuming high CPU and memory. It remains like this, when there are no external signals or active workflows. I have disabled the sticky workflow options so no workflow should be cached, instead the thread count is not going down. Is there any way to resolve this further ?
I was able to identify the issue. The issue was due to creation of multiple WorkflowClient per request. Each workflow client bootstraps its own tasklist and thread pool to dispatch tasks.
In java based applications, we should create single workflow clients for each flow that we want to orchestrate. For example I was creating workflows to orchestrate order shipments in an e-commerce application, so I created single workflow clients, one for forward journey and other for reverse. It worked like a charm.