AWS Beanstalk - Share Database layer between worker and webserver

Question

I am building some web application using NodeJS. I deployed the application to AWS beanstalk web server and I decoupled some heavy duty tasks, And deployed the heavy tasks to different AWS beanstalk workers, The main web server is executing the worker with SQS (aws queue).

The main web server has a big DB layer to connect, get and save to the database. The workers do need some db actions, every worker type need different actions that already implemented in the DB layer of the main web server.

So I'm trying to find a way to use the DB layer of the web server without duplicate it in every worker type (that's what I'm doing now).

What I thought is to decouple the DB layer from the main application and to deploy it to a web server to it self, but that's looks like a waste because I don't need to listen to requests in DB layer all the time, I need some on demand actions.

I attached a picture of how it looks:

• number 1 the main server (with full db layer)
• number 2 the queue
• number 3 the workers. (with duplicated db actions)

Is there any good solution to it?

Answer 1

How many save/load operation you are doing from the workers? How many workers are running at the same time? I think it's hard to suggest single solution without understanding how much workers in there, and how much data to be saved.

In general, I can think of multiple solutions actually:

1. You can open API on the web server, so once the worker is finished, it will call the web server. If you are having many workers it will not work, so you might need a workers in between (let's call it the "save worker"), so current workers will enqueue the "save job", then the "save worker" will call the web server API, this will guarantee that the web server is not overloaded with save operations.

2. You can create a duplicate instance of the web server (let call it "worker web server") which is for saving results from the workers, so workers will push the result to a new SQS, at this "worker web server" will be listening to the save requests in the SQS. You can control which mode the web server will work by setting environment variable for example.

3. You maybe can extract the DB layer into a package or library that's used in web server plus the workers as well, but you need to be careful that you don't open a lot of connections in parallel to the database or performing a lot of db operations at same time.