Why is my K8s network policy denying access when it should by matching the label

Question

I am trying to implement an egress network policy that allows egress to only specific pods which is not working as expected, I am unsure how to resolve this while maintaining the least privilege.

I have a PostgreSQL db (service, statefulset, pod) with the label networking/client: auth-api-postgresdb and I have a migrations job with the label networking/client: auth-api-postgresdb-migrations both of these have network policies.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: auth-api-postgres
spec:
  policyTypes:
    - Ingress
  podSelector:
    matchLabels:
      networking/client: auth-api-postgresdb
  ingress:
    - ports:
        - protocol: TCP
          port: 5432
      from:
        - podSelector:
            matchLabels:
              networking/client: auth-api-postgresdb-migrations

and

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: auth-api-postgresdb-migrations
spec:
  policyTypes:
    - Egress
  podSelector:
    matchLabels:
      networking/client: auth-api-postgresdb-migrations
  egress:
    - ports:
        - protocol: TCP
          port: 5432
      to:
        - podSelector:
            matchLabels:
              networking/client: auth-api-postgresdb

The db policy works as expected, but the migrations egress is not, in its current state it is not allowing outbound traffic on 5432 to the DB, however, if I update the podSelector to {} keeping the port restriction then it can reach the database.

It appears that the egress must be trying to access some other service or pod before connecting to the DB but I cannot work out what it is. Here is everything running in the namespace with and without the label filter before running the job. I have also tested a Debain image with PostgreSQL-client and this has the same issues, but I did find I could connect directly to the pod but still could not connect to the service.

$>k get all -n test -l networking/client=auth-api-postgresdb
NAME                                   READY   STATUS    RESTARTS   AGE
pod/auth-api-postgresdb-postgresql-0   2/2     Running   0          3m42s

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/auth-api-postgresdb-postgresql           ClusterIP   172.20.130.101   <none>        5432/TCP   3m42s
service/auth-api-postgresdb-postgresql-hl        ClusterIP   None             <none>        5432/TCP   3m42s
service/auth-api-postgresdb-postgresql-metrics   ClusterIP   172.20.134.229   <none>        9187/TCP   3m42s

NAME                                              READY   AGE
statefulset.apps/auth-api-postgresdb-postgresql   1/1     3m42s
$>k get all -n test
NAME                                   READY   STATUS    RESTARTS   AGE
pod/auth-api-postgresdb-postgresql-0   2/2     Running   0          4m13s

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/auth-api-postgresdb-postgresql           ClusterIP   172.20.130.101   <none>        5432/TCP   4m13s
service/auth-api-postgresdb-postgresql-hl        ClusterIP   None             <none>        5432/TCP   4m13s
service/auth-api-postgresdb-postgresql-metrics   ClusterIP   172.20.134.229   <none>        9187/TCP   4m13s

NAME                                              READY   AGE
statefulset.apps/auth-api-postgresdb-postgresql   1/1     4m13s

The error looks like this:

...
Opening connection to database 'identity' on server 'tcp://auth-api-postgresdb-postgresql:5432'
...
Npgsql.NpgsqlException (0x80004005): Failed to connect to 172.20.130.101:5432
...

For additional evidence, the namespace has a default deny policy a core DNS policy and an Istio proxy policy, although I disabled Istio injection to rule that out.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-access
spec:
  podSelector:
    matchLabels: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-istio-proxy-access
spec:
  podSelector:
    matchLabels: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: istio-system
          podSelector:
            matchLabels:
              app: istiod
      ports:
        - protocol: TCP
          port: 15012

Answer 1

Eventually, I found the issue with a lot of trial and error. It is not obvious and could be better documented on the network policy page. For a network policy to correctly match a pod via a service, the pod must have the correct labels as must the service, but the service must also have the match labels set in the spec.selector section.

Does not work:

apiVersion: v1
kind: Service
metadata:
  name: auth-api-postgresdb-postgresql-svc
  namespace: test
  labels:
    app: auth-api
    app.kubernetes.io/name: postgresql
    helm.sh/chart: postgresql-14.3.0
    networking/client: auth-api-postgresdb
  annotations:
    meta.helm.sh/release-name: auth-api-postgresdb
    meta.helm.sh/release-namespace: test
spec:
  selector:
    app: auth-api
    app.kubernetes.io/instance: auth-api-postgresdb
    app.kubernetes.io/name: PostgreSQL
...

Does work

apiVersion: v1
kind: Service
metadata:
  name: auth-api-postgresdb-postgresql-svc
  namespace: test
  labels:
    app: auth-api
    app.kubernetes.io/name: postgresql
    helm.sh/chart: postgresql-14.3.0
    networking/client: auth-api-postgresdb
  annotations:
    meta.helm.sh/release-name: auth-api-postgresdb
    meta.helm.sh/release-namespace: test
spec:
  selector:
    app: auth-api
    app.kubernetes.io/instance: auth-api-postgresdb
    app.kubernetes.io/name: postgresql
    networking/client: auth-api-postgresdb
...