istio-proxy not logging in-cluster traffic

Question

I have a basic istio1.12.2 setup on GKE 1.21.6-gke.1500, all the pods have istio-proxy injected. I have set meshConfig accessLogFile.

installed istio myself with

istioctl install -f istioOperator.yaml

# The default API gateway facing ingress gateway
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: default-installation
  annotations:
    config.kubernetes.io/local-config: "true"
spec:
  profile: default
  hub: gcr.io/istio-release
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    base:
      enabled: true
    pilot:
      enabled: true
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      label:
        app: istio-ingressgateway
        istio: ingressgateway

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "abc.com"

apiVersion: apps/v1
kind: Deployment
metadata:
  name: abc
  labels:
    app: abc
spec:
  selector:
    matchLabels:
      app: abc
  template:
    metadata:
      labels:
        app: abc
    spec:
      containers:
      - name: abc
        image: "abc"
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: abc
spec:
  selector:
    app: abc
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: virtual-abc-service
  namespace: default
spec:
  hosts:
    - abc.com
    - abc.default.svc.cluster.local
  gateways:
    - istio-system/istio-ingressgateway
    - mesh
  http:
    - route:
        - destination:
            host: abc.default.svc.cluster.local
            port:
              number: 80

I tried hitting the load balancer and in the ingressgateway pod I see the log (Due to some firewall I can't hit the load balancer but I port-forwarded it to 127.0.0.1:8080, therefore the 127.0.0.1, 404 is the expected response)

GET / HTTP/1.1" 404 - via_upstream - "-" 0 47 8 8 "10.32.4.9" "curl/7.77.0" "generatedlbname" "abc.com:8080" "10.32.0.24:8080" outbound|80||abc.default.svc.cluster.local 10.32.4.9:36756 127.0.0.1:8080 127.0.0.1:42412

10.32.4.9 is the ip of the ingressgateway

10.32.0.24 is the ip of the abc app pod

but when I checked the istio-proxy log for all the abc app, I don't see any logs for this call. I did get the response I expected so the call must have hit my app but somehow I'm not seeing logs in the pods.

is the request somehow not routed to istio-proxy? or are there any other settings I need to do.

Answer 1

The issue seems to be that I was hitting the wrong port. I saw the logs once I was hitting the correct port. although it's still weird for me that I'm not seeing logs by envoy, even though the port is not valid the pod ip is so I'd assume the packet passed through envoy