How to orchestrate subsets of Docker compose services

I am using a bunch of Docker containers to run a webserver + database + whatever else.

Currently I have each service defined in its own docker-compose.yml, file like:

project-root/docker/
  /base/docker-compose.yml 
  /postgres/docker-compose.yml
  /web-server/docker-compose.yml
  /web-app/docker-compose.yml
  /rabbitmq/docker-compose.yml
  /reverse-proxy/docker-compose.yml

...etc. Depending on the context (i.e. local development vs production), I might need to start one or more of the services at once. For local development I might want only postgres and web-server, but not reverse-proxy. Whereas in prod I might want all services running.

I've been using some different scripts, similar to:

# ./scripts/start-local.sh

docker compose /
  -f ./docker/base/docker-compose/yml /
  -f ./docker/service-a/docker-compose.yml /
  -f ./docker/sevice-b/docker-compose.yml /
  up

...to start the different subsets of services, but it feels messy. Often I get orphaned container messages, and other times --remove-orphans will kill a service that I'd rather have kept running.

Ideally, I'd like to be able to start some subset of services, and then add/remove other services as needed without Docker considering it a different "project" (and triggering the orphaned container stuff).

Is there a better approach to handling this sort of "run subset of all services" use case using the Docker CLI?

Solution

Here are couple of options:

Option 1: COMPOSE_FILE

Set COMPOSE_FILE variable to include the docker-compose files that you want (e.g. based on whether you're on dev/staging/prod).

I like to define a base file for core services which I always want, plus additional files for logical pieces which I may or may not want.

Example:

COMPOSE_FILE=compose.yaml:compose.dev.yaml:compose.mailcatcher.yaml:compose.traefik.yaml

I set this variable by sourcing a bash script (e.g. source docker.env). After that I can simply run docker compose up, etc.

Option 2: Use profiles

Use profiles (this is a newer compose feature). This allows you to keep all of your services in a single docker-compose file (though you may need a prod file to override some details). In each service, you can define a service to be part of one or more profiles:

services:
  frontend:
    image: frontend
    profiles: [core]

  phpmyadmin:
    image: phpmyadmin
    depends_on: [db]
    profiles: [dev]

  backend:
    image: backend
    profiles: [core]

Then you can start services for one or more profiles like this:

   docker compose --profile core up   # start one profile
   docker compose --profile core --profile dev up  # start multiple profiles

But to make it easy for myself, I set the COMPOSE_PROFILES variable in my script (sourcing it like above)

export COMPOSE_PROFILES="core dev"

so that I can simply do

   docker compose up

and it knows which profiles I want.