Shared istio egress gateway in multi-cluster/multi-primary mesh

We are trying to setup an egress gateway in a multi-cluster/multi-primary mesh configuration where the egress gateway is located in only one cluster but used from both.

diagram of desired setup

The use case is that the clusters are in different network zones and we want to be able to route traffic to one zone transparently to the clients in the other zone.

We followed this guide in one cluster and it worked fine. However we have trouble setting up the VirtualService in the second cluster to use the egress gateway in the first cluster.

When deploying the following virtual service to the second cluster we get 503 with cluster_not_found.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: direct-cnn-through-egress-gateway
spec:
  hosts:
  - edition.cnn.com
  gateways:
  - istio-egressgateway
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        port:
          number: 80
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      port: 80
    route:
    - destination:
        host: edition.cnn.com
        port:
          number: 80
      weight: 100

The endpoints proxy config on a pod in the second cluster misses the istio-egressgateway.istio-gateways.svc.cluster.local endpoints (all other services are discovered and directed to the eastwest gateway of the other cluster). We believe that this is the reason that this VirtualService doesn't work in the second cluster.

As a workaround we could redirect the egress traffic to the ingress gateway of the first cluster but this has the disadvantage that the traffic leaves and re-enters the mesh which probably has an impact on tracing and monitoring.

Is it currently possible to setup a single egress gateway that can be used by all clusters in the mesh or do we have to go with the workaround?

Solution

According to the comments the solution should works as below:

To create a multi-cluster deployment you can use this tutorial. In this situation cross cluster workload of normal services works fine. However, there is a problem with getting the traffic to the egress gateway routed via the eastwest gateway. This can be solved with this example. You should also change kind: VirtualService to kind: ServiceEntry in both clusters.

Like Tobias Henkel mentioned:

I got it to work fine with the service entry if I target the ingress gateway on ports 80/443 which then dispatches further to the mesh external services.

You can also use Admiral to automate traffic routing.