run a command in docker container during docker compose up

Question

I have a docker-compose.yaml for local airflow setup. It would setup all airflow services. In the service container airflow-webserver, I want to run a few commands which could be a ssh command or a airflow connections add ... command.

The airflow-webserver service from docker-compose.yaml is given below:

  airflow-webserver:
    ...
    command: webserver
    ports:
      - 8080:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    depends_on:
      ...
      airflow-init:
        condition: service_completed_successfully

The commands I want to run in this container is:

airflow connections add conn1 --conn-type 'mysql' --conn-login 'user' ...
ssh -o StrictHostKeyChecking=no -4 -N -L ...

I tried to utilize the command section command: webserver and used commnd: webserver && bash -c "... but I got following message:

airflow-webserver-1  | airflow command error: argument GROUP_OR_COMMAND: invalid choice: 'webserver &&' (choose from 'celery', 'cheat-sheet', 'config', 'connections', 'dag-processor', 
'dags', 'db', 'info', 'jobs', 'kerberos', 'kubernetes', 'plugins', 'pools', 'providers', 'roles', 'rotate-fernet-key', 'scheduler', 'standalone', 'sync-perm', 'tasks', 'triggerer', 'users', 'variables', 'version', 'webserver'), see help above.

Next option I tried is adding entrypoint as:

    command: webserver
    entrypoint: |
      bash -c "airflow connections add ...
    ...

but this option repeatedly gives me following error:

airflow-webserver-1  | Traceback (most recent call last):
airflow-webserver-1  |   File "/home/airflow/.local/bin/airflow", line 5, in <module>
airflow-webserver-1  |     from airflow.__main__ import main
airflow-webserver-1  | ModuleNotFoundError: No module named 'airflow'

Even the basic entrypoint: bash -c "echo ..." command gets executed repeatedly.

Since above options did not worked. I have been running following command manually after running docker compose up.

docker-compose exec -T airflow-webserver bash -c "
  airflow connections add ...
  ssh -o StrictHostKeyChecking=no ..."

All I wanted is to embed this command within docker-compose.yaml file to run it after airflow-webserver container is created.

Answer 1

I found a solution for running commands in a docker container while running docker compose up. For this, in my existing docker-compose.yaml file, I created a service with docker image with granted privileges. Then created a script and attached it to a volume. Then added the path of script in command section. Example:

  custom-script-runner:
    image: docker
    privileged: true
    working_dir: /opt/airflow
    volumes:
      - .:/opt/airflow
      - /var/run/docker.sock:/var/run/docker.sock
      - ./setup.sh:/opt/airflow/setup.sh
    command: /opt/airflow/setup.sh
    depends_on:
      airflow-webserver:
        condition: service_healthy

script.sh:

docker ps -a #check your running containers
docker exec -i myproject-airflow-webserver-1 bash -c "ssh -o ..."
docker exec -i myproject-airflow-webserver-1 bash -c "airflow connections add ..."

This link helped me to get the answer.