docker-compose healthcheck does not work in a way it is expected for making container a run first and then container B

Question

I am using docker compose to run couple of service which depends on each other. Here is part of the docker-compose:

  backend:
    build: .
    command: bash -c "npm run build && npm start"
    ports: 
      - "3015:3015"
    depends_on:
      - couchdb
      - redis
      - uds-mock-server
    volumes:
      - /app/node_modules
      - .:/app
    user: root
  api-test:
    restart: always 
    build: .
    depends_on:
      - backend
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3015/readiness"]
      interval: 200s
      timeout: 200s
      retries: 5
    user: root

As you see I have two service over there and backend should first run and the server needs to be ready then api-test can start. backend has an endpoint: localhost:2015/readiness and whenever it returns 200 then api test can start. When I run while building the order is respected so backend first followed by api-mock but when docker compose starts running them api-test run quicker and since it relies on the backend to be ready it fails.

Base on the following:

Docker Compose wait for container X before starting Y

and

Docker healthcheck in composer file

It is suggested that I should use healthcheck which I do in api test:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3015/readiness"]
  interval: 200s
  timeout: 200s
  retries: 5

If I get this correctly api-test should run and call the readiness endpoint and wait until it hears from backend readiness (up to 200s) if it fails it waits for 200s and then try again for 5 times. But what I see is api-test keep failing and restarting and does not even give a chance to backend to run and it keeps doing this like a loop. Am I missing anything? Any help is really appreciated

Answer 1

The two examples are based on the condition form of depends_on which is no longer supported in compose version 3. So, unless your docker-compose version is <3 the healthcheck will not help you much. The healthcheck sets the status of the container (starting, healthy or unhealthy) but docker-compose does not wait until backend container is healthy before starting the app-test. There is a detailed explanation about how depends_on works in Control startup and shutdown order in Compose

As a side note, the healthcheck in your compose file sets the status of the app-test container and not backend.

Therefore to control when the api-test can start, you have to wrap the service command of the container. For your particular case the following will do the job:

bash -c 'while [[ "$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' https://backend:3015/readiness)" != "200" ]]; do echo ..; sleep 5; done; echo backend is up; <service_command>'

It tries to connect to backend every 5 seconds (the connection timeout is 2s). When the received HTTP status code is 200 OK the loop ends and it executes the <service_command>

The relevant docker-compose part:

  api-test:
    restart: always
    command: bash -c 'while [[ "$$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' uds-mock-server:4000/readiness)" != "200" ]]; do echo ..; sleep 5; done; echo backend is up;npm start'
    depends_on:
      - backend
      ...

Hope this helps.