Search code examples
dockerairflow

Airflow: Could not read served logs: [Errno -3] Temporary failure in name resolution

How can you set the public hostname of an airflow worker in docker while using docker-compose?

I've tried ..

  • Setting the HOSTNAME environment variable
  • Setting the hostname field in the docker-compose
  • Updating the deployment to celery worker -H ${HOSTNAME}

The celery worker name did come up correctly as the hostname in flower, but the instance details still show the docker Container ID instead of the hostname. Since airflow is using the wrong hostname the UI cannot access the logs of workers run on a separate docker server.

 airflow-worker:
    <<: *airflow-common
    command: celery worker -H ${HOSTNAME}
    hostname: ${HOSTNAME}
    healthcheck:
      test:
        - "CMD-SHELL"
        - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    environment:
      <<: *airflow-common-env
      # Required to handle warm shutdown of the celery workers properly
      # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
      DUMB_INIT_SETSID: "0"
      HOSTNAME: ${HOSTNAME}
    restart: always

This is using Airflow Version: v2.6.3

I believe this is where the hostname need to be changed, in the instance details

enter image description here

Solution

  • In your environment variables file you shoud define HOSTNAME=<Server IP/Domain mapped with>. I did set 'HOSTNAME' in env file to the 'Server IP' value and it worked perfectly, especially in multi-workers deployment architecture.