Search code examples
microservices

Avoid bottlenecks in microservices

I'm going to apply Microservices for my Datawarehouse application. There are 4 main Microservices in application:

1) Data Service: Import/Export external data sources to DWH and Query data from DWH.

2) Analytics Service: for chart visualization on UI

3) Machine Learning: for recommendation system

4) Reports: for report generating

The diagram as below:

enter image description here

Each service has its own DB and they communicate directly with each other via TCP and Thift serialization. The problem here is Data Service suffer a high load from other services and can become a SPOF of application. Data in DWH is big too (maybe up to hundred miliions of records). How to avoid the bottlenecks for Data Service in this case? Or How do I define a properly bounded context to avoid the bottlenecks?

Solution

  • You may think about

    • splitting Data Service into few microservices, based on some business logic;
    • modify Data Service (if needed) to support more than one instance of service. Then use the load balancer to split requests between those instances.

    A load balancer is a device that acts as a reverse proxy and distributes network or application traffic across a number of servers. Load balancers are used to increase capacity (concurrent users) and reliability of applications.

    Regarding "One database, multiple services":

    Each microservice need to have own data storage, otherwise, you do not have a decomposition. If we are talking about relation database, then this can be achieved using one of the following patterns:

    • Private tables per Service – each service owns a set of tables that must only be accessed by that service
    • Schema perService – each service has a database schema that’s private to that service
    • Database per Service – each service has it’s own database.

    If your services using separate tables from Data Warehouse database and Data Service only provides access layer to database without any additional processing logic, then yes, you may remove Data Service and move data access logic to corresponding services. But think on another hand - right now you have only one place (Data Service), that knows how to access and manipulate with Data Warehouse that is what microservices are about.