Communicate with downstream envoy behind private network from Upstream Envoy

Question

I just opened this issue on envoy but feel the stackOverflow community can add value.

See Diagram for better understanding:

Also See the link to wireguard achieving something similiar below:

Title: Is it possible from the downstream envoy to establish a connection with the upstream, from the upstream service to send a tcp request to a service behind the downstream envoy without exposing the downstream envoy to a public IP/Port.

Description: In my scenario, I have a central envoy (Upstream) that contains a management service. This management service is meant to connect to multiple clusters. Each cluster is behind a private network with the only option engress with no ingress option for envoy.

This StackOverflow describes the senario very well:

For a hybrid-cloud use-case we are looking into the suitability of EnvoyProxy to act as a solution to move data across an on-premise firewall. The intended setup is as follows:

App A is located in an on premise network with no direct outbound or inbound Internet connection

App B is located on the cloud

An envoy proxy (PC) is placed in the cloud

An envoy proxy (PA) is placed in the on-premise network, and configured to allow outbound network connection to PC

PA creates a open bi-directional authenticated TLS connection to PC, effectively creating a tunnel between them

App B invokes an API endpoint at PC, which gets routed to PA over the open TLS connection, and forwarded by PA to App A

Routing data with Envoy to upstream clusters is well documented. However, we are wondering if Envoy is able to setup a TLS connection between two proxy instances and use that channel in both directions. The constraint is that this TLS connection can be setup from one direction (outbound) only.

We want the management service to make requests to applications over HTTP or TCP.

I have been looking into the TCPProxy option that is described in the video in stackOverflow but I am struggling to find the target needed to communicate with the downstream server.

I have also been following the examples in the Sandbox documentation but my issue is that I am not looking to setup a streaming connection but a way to tunnel the two networks so their accessable.

I have been struggling with the understanding due to limited explaination in the docs.

Things I have looked into are and tried a few of the sandboxes examples they give.

WebSocket is a one to one streaming and does not allow reverse proxy to multiple services.

I have read that Envoy Connect/TCPProxy could be a solution, I tried this but struggled to find an endpoint to the DownStream Cluster that is connected to the listener on the Upstream server.

TCP Proxy The TCP proxy filter performs basic 1:1 network connection proxy between downstream clients and upstream clusters. It can be used by itself as an stunnel replacement

This explains its possible. but how?

Any advise would be highly apprichiated.

[optional Relevant Links:]

StackOverflow

GitHub Issue #22697

Wireguard Client Communication

Answer 1

It turns out that envoy does not support a multi direction tunnel.

Alternative is a solution like Teleport.

Details in the linked Github issue page