How to use a separate container service with each containerised celery worker?

Question

I am using docker-compose to run multiple celery workers and struggling to make workers use this zeta0/alpine-tor rotating proxy pool image the way I want.

Currently my docker-compose.yml looks like this:

version: '3'
services:
    worker:
        build:
            context: .
            dockerfile: dockerfile
        volumes:
            - type: bind
              source: .
              target: /app
        links:
            - rabbit
            - tor
        depends_on:
            - rabbit
            - tor
    rabbit:
        hostname: rabbit
        image: rabbitmq:latest
        environment:
           - RABBITMQ_DEFAULT_USER=myuser
           - RABBITMQ_DEFAULT_PASS=mypassword
        ports:
            - "5672:5672"
    tor:
        hostname: tor
        image: zeta0/alpine-tor
        environment:
            - tors=25
            - privoxy=1
            - new_circuit_period=10
        ports:
            - "8118:8118"
            - "2090:2090"

And then I run: docker-compose up --scale worker=5.

The problem is that all workers are using the same tor service. I want to have a separate alpine-tor service for every worker. Is this possible to do with docker-compose? If so, what should I change to achieve that?

Answer 1

You could try running the worker as 5 different docker-compose services(but all using the same worker image), and also running the tor service as 5 different docker-compose services(again, using the same tor image).

For each of the 5 worker services, you could pass in an environment variable indicating which tor service you want the worker to communicate with. Then in your worker, you can use this environment variable in the tor URL.

So your docker-compose.yml would look like this(some config removed for brevity):

version: '3'
services:
    worker_1:
        environment:
            - TOR_SERVICE=tor_1
        ...
    worker_2:
        environment:
            - TOR_SERVICE=tor_2
        ...
    worker_3:
        environment:
            - TOR_SERVICE=tor_3
        ...
    worker_4:
        environment:
            - TOR_SERVICE=tor_4
        ...
    worker_5:
        environment:
            - TOR_SERVICE=tor_5
        ...
    rabbit:
        hostname: rabbit
        image: rabbitmq:latest
        environment:
           - RABBITMQ_DEFAULT_USER=myuser
           - RABBITMQ_DEFAULT_PASS=mypassword
        ports:
            - "5672:5672"
    tor_1:
        ...
    tor_2:
        ...
    tor_3:
        ...
    tor_4:
        ...
    tor_5:
        ...