iptables in kuberntes init container does't work

Question

Background:

I'm trying to use goreplay to mirror the traffic to other destination. I found that k8s service is a load balancing on layer 4 which cause the traffic can not be capture by goreplay,So i decide to add a reverse-proxy sidecar inside pod just like istio does.

Here is my pod yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx
    ports:
    - containerPort: 80
      name: http
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: proxy
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 10m
        memory: 40Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File  
    volumeMounts:
      - mountPath: /etc/nginx/conf.d
        name: default   
  initContainers:
    - command: 
      - iptables
      args:
      - -t 
      - nat
      - -A
      - PREROUTING
      - -p
      - tcp
      - --dport
      - "80"
      - -j
      - REDIRECT
      - --to-ports
      - "15001"
      image: soarinferret/iptablesproxy
      imagePullPolicy: IfNotPresent
      name: istio-init
      resources:
        limits:
          cpu: 100m
          memory: 50Mi
        requests:
          cpu: 10m
          memory: 10Mi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          add:
          - NET_ADMIN
          - NET_RAW
          drop:
          - ALL
        privileged: false
        readOnlyRootFilesystem: false
        runAsGroup: 0
        runAsNonRoot: false
        runAsUser: 0
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File        
  dnsPolicy: ClusterFirst
  terminationGracePeriodSeconds: 30
  volumes:
    - configMap:
        defaultMode: 256
        name: default
        optional: false
      name: default
---

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

--- 

apiVersion: v1
data:
  default.conf: |
    server {
        listen       15001;
        server_name  localhost;
        access_log  /var/log/nginx/host.access.log  main;
        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }
kind: ConfigMap
metadata:
  name: default
  namespace: default

I use kubectl port-forward service/nginx 8080:80 and then curl http://localhost:8080,the traffic were sent directly to nginx not my proxy.

WHAT I WANT:

1. A way to let goreplay to capture traffic that load balanced by k8s service.
2. Correct iptables rule to let traffic success route to my proxy sideCar.

Thanks for any help!

Answer 1

As @Jonyhy96 mentioned in comments the only things which need to be changed here is to the privileged value to true in the securityContext field of initContainer.

Privileged - determines if any container in a pod can enable privileged mode. By default a container is not allowed to access any devices on the host, but a "privileged" container is given access to all devices on the host. This allows the container nearly all the same access as processes running on the host. This is useful for containers that want to use linux capabilities like manipulating the network stack and accessing devices.

---

So the initContainer would look like this

initContainers:
    - command: 
      - iptables
      args:
      - -t 
      - nat
      - -A
      - PREROUTING
      - -p
      - tcp
      - --dport
      - "80"
      - -j
      - REDIRECT
      - --to-ports
      - "15001"
      image: soarinferret/iptablesproxy
      imagePullPolicy: IfNotPresent
      name: istio-init
      resources:
        limits:
          cpu: 100m
          memory: 50Mi
        requests:
          cpu: 10m
          memory: 10Mi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          add:
          - NET_ADMIN
          - NET_RAW
          drop:
          - ALL
        privileged: true   <---- changed from false
        readOnlyRootFilesystem: false
        runAsGroup: 0
        runAsNonRoot: false
        runAsUser: 0
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File 

---

There is very good tutorial about that, not exactly on nginx, but explains how to actually build the proxy.