Search code examples
multithreadingspring-bootarchitectureautoscalingsystem-design

Is it a good practice to use multithreading to handle requests in bulk in a micro services architecture systems?

  1. Requirement:

I have to design a micro service which performs search query in a sql db multiple times(say 7 calls) along with multiple third party http calls(say 8 calls) in sequential and interleaved manner to complete an order, by saying sequential I mean before next call of DB or third party previous call to DB or third party must be completed as the result of these calls will be used in further third party or search operations in DB.

  1. Resources:

    I) CPU: 4 cores(per instance)

    II) RAM: 4 GB(per instance)

    III) It can be auto scaled upto at max of 4 pods or instances.

    IV) Deployment: Open Shift (Own cloud architecture)

    V) Framework: Spring Boot

  2. My Solution:

I've created a fixed thread pool of 5 threads(Size of blocking queue is not configured, also there are another 20 fixed pool threads running apart from these 5 threads for creating orders of multiple types i.e. in total there are 25 threads running per instance) using thread pool executor of Java. So when multiple requests are sent to this micro service I keep submitting the job and the JVM by using some scheduling algorithms schedules these jobs and complete the jobs.

  1. Problem:

I'm not able to achieve the expected through put, using above approach the micro service is able to achieve only 3 to 5 tps or orders per second which is very low. Sometimes it also happens that tomcat gets choked and we have to restart services to bring back the system in responsive situation.

  1. Observation:

I've observed that even when orders are processed very slowly by the thread pool executor if I call orders api through jmeter at the same time when things are going slow, these kind of requests which are directly landing on the controller layer are processed faster than the request getting processed by thread pool executor.

  1. My Questions

    I) What changes I should make at the architectural level to make through put upto 50 to 100 tps.

    II) What changes should be done so that even if traffic on this service increases in future then the service can either be auto scaled or justification to increase hardware resources can be given easily.

    III) Is this the way tech giants(Amazon, Paypal) solve scaling problems like these using multithreading to optimise performance of their code.

You can assume that third parties are responding as expected and query optimisation is already done with proper indexing.

Solution

  • Tomcat already has a very robust thread pooling algorithm. Making your own thread pool is likely causing deadlocks and slowing things down. The java threading model is non-trivial, and you likely are causing more problems than you are solving. This is further evidenced by the fact that you are getting better performance relying on Tomcat's scheduling when you hit the controller directly.

    High-volume services generally solve problems like this by scaling wide, keeping things as stateless as possible. This allows you to allocate many small servers to solve the solution much more efficiently than a single large server.

    Debugging multi-threaded executions is not for the faint of heart. I would highly recommend you simplify things as much as possible. The most important bit about threading is to avoid mutable state. Mutable state is the bane of shared executions, moving memory around and forcing reads through to main memory can be very expensive, often costing far more than savings due to threading.

    Finally, the way you are describing your application, it's all I/O bound anyway. Why are you bothering with threading when it's likely I/O that's slowing it down?