Search code examples
loggingkuberneteshttp-headersistioenvoyproxy

Enable http header logging for envoy in istio

I want to be able to capture (log) (at least some of) envoy's HTTP headers on my istio service mesh.

I have gone through envoy's docs, and in the log levels' section, it does not mention any header-specific information.

Currently, my istio-proxy log is like this (this is from a stern output):

mysvc-69c46fbc75-d9v8j istio-proxy {"bytes_sent":"124","upstream_cluster":"inbound|80|http|mysvc.default.svc.cluster.local","downstream_remote_address":"10.11.11.1:0","authority":"some.url.com","path":"/health?source=dd_cluster_agent","protocol":"HTTP/1.1","upstream_service_time":"1","upstream_local_address":"127.0.0.1:40406","duration":"2","upstream_transport_failure_reason":"-","route_name":"default","downstream_local_address":"10.11.32.32:20000","user_agent":"Datadog Agent/7.24.0","response_code":"200","response_flags":"-","start_time":"2021-01-17T18:54:57.449Z","method":"GET","request_id":"61ae63c7-aa10-911b-9562-939kdhd49ddhj","upstream_host":"127.0.0.1:20000","x_forwarded_for":"10.16.32.1","requested_server_name":"outbound_.80_.mysvc_.faros.default.svc.cluster.local","bytes_received":"0","istio_policy_status":"-"}

Is there a way to log http headers? (ideally some of them, to keep the logging cost under control)

edit1 following advice in the comments, I checked my istio-operator resource and I see that access logging seems to be enabled

    meshConfig:
      accessLogEncoding: JSON
      accessLogFile: /dev/stdout

edit2 I have also tried the following:

curl -i -H "Custom-Header: application/json" https://my.url.net

but in the logs of the istio-ingressgateway I don't see my custom header

istio-ingressgateway-58f69d8696-rmpwn istio-proxy {"user_agent":"curl/7.64.1","response_code":"200","response_flags":"-","start_time":"2021-01-18T19:02:48.645Z","method":"GET","request_id":"8e32c93c-484d-9c56-9489-8c5392793d97","upstream_host":"10.16.32.55:20000","x_forwarded_for":"10.16.32.1","requested_server_name":"my.url.net","bytes_received":"0","istio_policy_status":"-","bytes_sent":"124","upstream_cluster":"outbound|80||mysvc.default.svc.cluster.local","downstream_remote_address":"10.16.32.1:52804","authority":"my.url.net","path":"/","protocol":"HTTP/2","upstream_service_time":"9","upstream_local_address":"10.16.32.17:49826","duration":"10","upstream_transport_failure_reason":"-","route_name":"-","downstream_local_address":"10.16.32.17:8443"}
Solution

  • I think I've succesfully made a reproduction of your issue and I was able to print MY_CUSTOM_HEADER in the ingress gateway logs.

    There is a part of my istio ingress gateway logs.

    [2021-01-20T08:26:18.587Z] pkarambol GET /productpage HTTP/1.1 200

    I've used below curl command:

    curl -v -H "MY_CUSTOM_HEADER: pkarambol" xx.xxx.xx.xxx/productpage

    To make that happen you have to change the default format of the logs. As mentioned in the documentation, you can use meshConfig.accessLogFormat to change that.

    There is an example of Istio Operator I've used.

    %REQ(MY_CUSTOM_HEADER)% is a part responsible for displaying the custom header.

    apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: demo
  meshConfig:
    accessLogFile: /dev/stdout
    accessLogFormat: "[%START_TIME%] %REQ(MY_CUSTOM_HEADER)% %REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%
%RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION%
%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% %REQ(X-FORWARDED-FOR)% %REQ(USER-AGENT)%
%REQ(X-REQUEST-ID)% %REQ(:AUTHORITY)% %UPSTREAM_HOST%\n"