Cannot apply inter-pod affinity to Airflow scheduler

Question

I am encountering a weird behavior when I try to attach podAffinity to the Scheduler deployment from the official Airflow helm chart, like:

  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app 
            operator: In
            values:
            - postgresql
        topologyKey: "kubernetes.io/hostname"

With an example Deployment to which the podAffinity should "hook up" to:

metadata:
  name: {{ template "postgresql.fullname" . }}
  labels:
    app: postgresql
    chart: {{ template "postgresql.chart" . }}
    release: {{ .Release.Name | quote }}
    heritage: {{ .Release.Service | quote }}
spec:
  serviceName: {{ template "postgresql.fullname" . }}-headless
  replicas: 1
  selector:
    matchLabels:
      app: postgresql
      release: {{ .Release.Name | quote }}
  template:
    metadata:
      name: {{ template "postgresql.fullname" . }}
      labels:
        app: postgresql
        chart: {{ template "postgresql.chart" . }}

Which results in:

NotTriggerScaleUp: pod didn't trigger scale-up: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod affinity rules

However, applying the same podAffinity config to the Webserver deployment works just fine. Plus, changing the example Deployment to a vanilla nginx manifested itself in the outcome.

It does not seem to be any resource limitation issue since I already tried various configs, every time with the same result. I do not use any custom configurations apart from node affinity.

Has anyone encounter the same or has any idea what I might do wrong?

Setup:

• AKS cluster
• Airflow helm chart 1.1.0
• Airflow 1.10.15 (but I don't think this matters)
• kubectl client (1.22.1) and server (1.20.7)

Links to Airflow charts:

• Scheduler
• Webserver

Answer 1

I've recreated this scenario on my GKE cluster and I've decided to provide a Community Wiki answer to show that the podAffinity on the Scheduler works as expected. I will describe step by step how I tested it below.

---

1. In the values.yaml file, I've configured the podAffinity as follows:

$ cat values.yaml
...
# Airflow scheduler settings
scheduler:  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - postgresql
        topologyKey: "kubernetes.io/hostname"
...

2. I've installed the Airflow on a Kubernetes cluster using the Helm package manager with the values.yaml file specified.

$ helm install airflow apache-airflow/airflow --values values.yaml

After a while we can check the status of the scheduler:

$ kubectl get pods -owide | grep "scheduler"
airflow-scheduler-79bfb664cc-7n68f   0/2     Pending   0          8m6s   <none>      <none>                                 <none>           <none>

3. I've created an example Deployment with the app: postgresql label:

$ cat test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: postgresql
  name: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgresql
  template:
    metadata:
      labels:
        app: postgresql
    spec:
      containers:
      - image: nginx
        name: nginx
        
$ kubectl apply -f test.yaml
deployment.apps/test created

$ kubectl get pods --show-labels | grep test
test-7d4c9c654-7lqns                 1/1     Running   0          2m   app=postgresql,...

4. Finally, we can check that the scheduler was successfully created:

$ kubectl get pods -o wide | grep "scheduler\|test"
airflow-scheduler-79bfb664cc-7n68f   2/2     Running   0          14m     10.X.1.6    nodeA     
test-7d4c9c654-7lqns                 1/1     Running   0          2m27s   10.X.1.5    nodeA

---

Additionally, detailed informtion on pod affinity and pod anti-affinity can be found in the Understanding pod affinity documentation:

Pod affinity and pod anti-affinity allow you to constrain which nodes your pod is eligible to be scheduled on based on the key/value labels on other pods.

Pod affinity can tell the scheduler to locate a new pod on the same node as other pods if the label selector on the new pod matches the label on the current pod.

Pod anti-affinity can prevent the scheduler from locating a new pod on the same node as pods with the same labels if the label selector on the new pod matches the label on the current pod.