Search code examples
kubernetesazure-akskubernetes-networkpolicycilium

NetworkPolicy does not allow egress on HTTP/HTTPS ports

I am using the NetworkPolicy below to allow egress on HTTP and HTTPS ports, but running wget https://google.com doesn't work when the network policy is applied. The domain name is resolved (DNS egress rule works) but connecting to the external host times out.

I've tried on minikube with cilium and on Azure with azure-npm in case it was some quirk with the network policy controller, but it behaves the same on both. I'm confused since I use the same method for DNS egress (which works) but this fails for other ports.

What's preventing egress on HTTP/HTTPS ports?

Kubernetes version 1.11.5

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  name: my-netpolicy
spec:
  egress:
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
  - ports:
    - port: 443
      protocol: UDP
    - port: 443
      protocol: TCP
    - port: 80
      protocol: UDP
    - port: 80
      protocol: TCP
  podSelector:
    matchLabels:
      my-label: my-app

(Yes, the UDP rules are probably unnecessary, but trying everything here)

(I've also tried wget on a private server in case Google/etc. block Azure IPs, same result)

(I've also tried matching ingress rules because "why not", same result)

kubectl describe on the network policy:

Name:         my-netpolicy
Namespace:    default
Created on:   2019-01-21 19:00:04 +0000 UTC
Labels:       ...
Annotations:  <none>
Spec:
  PodSelector:     ...
  Allowing ingress traffic:
    To Port: 8080/TCP
    From: <any> (traffic not restricted by source)
    ----------
    To Port: https/UDP
    To Port: https/TCP
    To Port: http/TCP
    To Port: http/UDP
    From: <any> (traffic not restricted by source)
  Allowing egress traffic:
    To Port: 53/UDP
    To Port: 53/TCP
    To: <any> (traffic not restricted by source)
    ----------
    To Port: https/UDP
    To Port: https/TCP
    To Port: http/UDP
    To Port: http/TCP
    To: <any> (traffic not restricted by source)
  Policy Types: Ingress, Egress

Minimal reproducible example:

apiVersion: v1
kind: Pod
metadata:
  name: netpolicy-poc-pod
  labels:
    name: netpolicy-poc-pod
spec:
  containers:
  - name: poc
    image: ubuntu:18.04
    command: ["bash", "-c", "while true; do sleep 1000; done"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: netpolicy-poc
spec:
  podSelector:
    matchLabels:
      name: netpolicy-poc-pod
  egress:
  - ports:
    - port: 80
      protocol: UDP
    - port: 80
      protocol: TCP
    - port: 443
      protocol: UDP
    - port: 443
      protocol: TCP
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
  ingress: []

Then:

kubectl exec -it netpolicy-poc /bin/bash
apt update
apt install wget -y
wget https://google.com
Solution

  • Turns out the policy I gave works fine, it's just that the controllers implementing the policy had some bugs. On Minikube+Cilium it just didn't work for IPv6 but worked fine for IPv4, and on AKS the feature is still generally in beta and there are other options that we could try. I haven't found anything on my specific issue when using the azure-npm implementation but since it works fine in Minikube on IPv4 I'll assume that it would work fine in Azure as well once a "working" controller is set up.

    Some resources I found for the Azure issue: