I have a project with the following structure:
backend
|-app # fastapi and celery
|-scrapper # selenium and celery
I am using celery to run long and short tasks, so I will have multiple queues/workers. Also, I will have tasks called from app
, but that will be processed by scrapper
. I am thinking how to split things among containers and I am not sure how to proceed. Here is what I am thinking to do:
I am kinda new to docker and this seems to be a waste of resources (to have multiple copies of the same thing, one to run fastapi and another to run celery alone), but if I understood right this is the way to go. I that right?
I would actually have four different containers:
Reason for this is simply the ability to manage and scale easily any piece of your infrastructure. At some point you may find that you do not have enough workers to handle heavy fastapi load - then you would just add more fastapi-celery
containers, etc.