k8s deletes nodeAffinity from pod specs

Question

Our system runs on GKE in a VPC-native network. We've recently upgraded from v1.9 to v1.21, and when we transferred the configuration, I've noticed the spec.template.spec.affinity.nodeAffinity in out kube-dns deployment is deleted and ignored. I tried manually adding this with "kubectl apply -f kube-dns-deployment.yaml"

I get "deployment.apps/kube-dns configured", but after a few seconds the kube-dns reverts to a configuration without this affinity.

This is the relevant code in the yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
  name: kube-dns
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-dns
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        components.gke.io/component-name: kubedns
        prometheus.io/port: "10054"
        prometheus.io/scrape: "true"
        scheduler.alpha.kubernetes.io/critical-pod: ""
        seccomp.security.alpha.kubernetes.io/pod: runtime/default
      creationTimestamp: null
      labels:
        k8s-app: kube-dns
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - preference:
              matchExpressions:
              - key: cloud.google.com/gke-nodepool
                operator: In
                values:
                - pool-1
            weight: 20
          - preference:
              matchExpressions:
              - key: cloud.google.com/gke-nodepool
                operator: In
                values:
                - pool-3
                - training-pool
            weight: 1
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: cloud.google.com/gke-nodepool
                operator: In
                values:
                - pool-1
                - pool-3
                - training-pool
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: k8s-app
                  operator: In
                  values:
                  - kube-dns
              topologyKey: kubernetes.io/hostname
            weight: 100
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: k8s-app
                operator: In
                values:
                - kube-dns
            topologyKey: cloud.google.com/hostname
      containers:
      ....
      dnsPolicy: Default
      nodeSelector:
        kubernetes.io/os: linux

This is what I get when I run $ kubectl get deployment kube-dns -n kube-system -o yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    ....
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
  name: kube-dns
  namespace: kube-system
  resourceVersion: "16650828"
  uid: ....
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-dns
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        components.gke.io/component-name: kubedns
        prometheus.io/port: "10054"
        prometheus.io/scrape: "true"
        scheduler.alpha.kubernetes.io/critical-pod: ""
        seccomp.security.alpha.kubernetes.io/pod: runtime/default
      creationTimestamp: null
      labels:
        k8s-app: kube-dns
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: k8s-app
                  operator: In
                  values:
                  - kube-dns
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      ...
      dnsPolicy: Default
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 65534
        supplementalGroups:
        - 65534
      serviceAccount: kube-dns
      serviceAccountName: kube-dns
      terminationGracePeriodSeconds: 30
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - key: components.gke.io/gke-managed-components
        operator: Exists
      volumes:
      - configMap:
          defaultMode: 420
          name: kube-dns
          optional: true
        name: kube-dns-config
status:
  ...

As you can see, GKE just REMOVES the NodeAffinity part, as well as one part of the podAffinity.

Answer 1

kube-dns is a service discovery mechanism within GKE, and the default DNS provider used by the clusters. It is managed by Google and that is why the changes are not holding, and most probably that part of the code was removed in the new version.

If you need to apply a custom configuration, you can do that following the guide Setting up a custom kube-dns Deployment.