Running Airflow on DC/OS using Docker

Question

I have setup DC/OS locally using their vagrant image, and I want to use DC/OS to run a Docker image containing Airflow, such that I can run the basic Hello World in the Airflow tutorial.

This is a fairly niche use case, and there aren't any guides or chatter online about the actual implementation of this.

In simple terms, how should I go about doing this?

Answer 1

You go to Services tab, click on "+" and add a JSON definition of a task:

{
  "id": "/airflow",
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "volumes": [],
    "docker": {
      "image": "puckel/docker-airflow"
    },
    "portMappings": [
      {
        "containerPort": 8080,
        "hostPort": 0,
        "protocol": "tcp"
      },
      {
        "containerPort": 5555,
        "hostPort": 0,
        "protocol": "tcp"
      },
      {
        "containerPort": 8793,
        "hostPort": 0,
        "protocol": "tcp"
      }
    ]
  },
  "mem": 1024,
  "requirePorts": false,
  "networks": [
    {
      "mode": "container/bridge"
    }
  ],
  "healthChecks": [],
  "fetch": [],
  "constraints": [],
  "cpus": 1,
  "labels": {
    "DCOS_SERVICE_PORT_INDEX": "0"
  }
}

which will spin up a this Docker image. When the service starts you should be able to click on the icon next to service name and navigate to Airflow UI (/service/airflow).

Note: I have zero experience with Airflow, but this is a general approach how you could start any task. You'll definitely need to configure database etc.