Istio: Egress gateway - use istio_requests_total metric

Question

I am new to Istio and I have following problem. I am trying to set up configuration of egress gateway for external service communicating through tls/443 like for the following example: https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway/#egress-gateway-for-https-traffic.

Everything seems to work correctly. For outbound traffic for 'externalapi' service, I am getting istio_tcp_connections_closed_total metric. And here's my question:

Is there any way to replace istio_tcp_connections_closed_total metric with istio_requests_total for outbound traffic going through egress gateway? I would like to get some additional information like response codes for outgoing traffic.

Here's my configuration:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: externalapi-egress
spec:
  hosts:
  - externalapi.mydomain.com
  ports:
  - number: 443
    name: tls
    protocol: TLS
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: externalapi-egress
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 443
      name: tls
      protocol: TLS
    hosts:
    - externalapi.mydomain.com
    tls:
      mode: PASSTHROUGH
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: externalapi-egress
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: externalapi-egress
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: externalapi
spec:
  hosts:
    - externalapi.mydomain.com
  gateways:
    - externalapi-egress
    - mesh
  tls:
  - match:
    - gateways:
      - mesh
      port: 443
      sniHosts:
        - externalapi.mydomain.com
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: externalapi-egress
        port:
          number: 443
      weight: 100
  - match:
    - gateways:
      - externalapi-egress
      port: 443
      sniHosts:
        - externalapi.mydomain.com
    route:
    - destination:
        host: externalapi.mydomain.com
        port:
          number: 443
      weight: 100

Other configuration information:

1. Istio: 1.8.0 installed through IstioOperator.

Thank you for helping me with this, Robert

Edit: I would like to find a way to have istio_requests_total metric generated for the traffic going to externalapi instead of istio_tcp_connections_closed_total.

Answer 1

tldr: you cannot do this.

---

Now the long answer.

From istio documentnion about metrics:

For HTTP, HTTP/2, and GRPC traffic, Istio generates the following metrics:

Request Count (istio_requests_total): This is a COUNTER incremented for every request handled by an Istio proxy.

. . .

For TCP traffic, Istio generates the following metrics:

Tcp Byte Sent (istio_tcp_sent_bytes_total): This is a COUNTER which measures the size of total bytes sent during response in case of a TCP connection.

Tcp Byte Received (istio_tcp_received_bytes_total): This is a COUNTER which measures the size of total bytes received during request in case of a TCP connection.

Tcp Connections Opened (istio_tcp_connections_opened_total): This is a COUNTER incremented for every opened connection.

Tcp Connections Closed (istio_tcp_connections_closed_total): This is a COUNTER incremented for every closed connection.

. . .

Notice that istio_requests_total (according to documentation) counts number of requests and this metric is available only for HTTP, HTTP/2, and GRPC traffic.

For TCP traffic there is no requests_total mertic because it would be hard to say what to define as a request. That is why for tcp you can only count bytes and number of connections.

Now you may say: "hey, I am not using tcp, I am using https (http over tls) so it should be able to count the requests, right?" - and you would be wrong.

Before I go further, let me first mention about "HTTP persistent connection" which is defined by wikipedia as:

HTTP persistent connection, also called HTTP keep-alive, or HTTP connection reuse, is the idea of using a single TCP connection to send and receive multiple HTTP requests/responses, as opposed to opening a new connection for every single request/response pair. The newer HTTP/2 protocol uses the same idea and takes it further to allow multiple concurrent requests/responses to be multiplexed over a single connection.

Now, why am I mentioning this?

TLS is encrypted traffic. Nothing can peek inside. In case your application is sending/receiving multiple requests/responses over single tls connection (using HTTP persistent connection), it's impossible to count every consecutive request because it is end-to-end encrypted.