Search code examples
envoyproxy

Envoy dynamic routing

Is it possible to write an envoy plugin that would resolve request destination at runtime?

Problem I'm trying to solve:

  1. requests arrive at xxxx.foo.com where cardinality of xxxx is high (there is wildcard DNS entry)
  2. requests are authorized
  3. there is in memory dynamic database that maps xxxx values to internal IP:port pair that request must be forwarded to. Mappings change frequently.

What is the easiest way to to have envoy route requests to correct destination?

Solution

  • I ended up using a combination of xDS protocol and external auth endpoint.

    1. Envoy is configured to get cluster and route configuration via xDS from control plane. See: https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol
    2. envoy.filters.http.ext_authz filter is added so that Envoy would make a gRPC request for each incoming HTTP request.
    3. A "blackhole" cluster is configured that doesn't correspond to any real service.
    4. A wildcard route is configured that routes all requests to the "blackhole" cluster.

    When HTTP request arrives, Envoy has to make a routing decision immediately. At this point is too late to add new routes and cluster definitions so following trick is used:

    1. Envoy routes the request to "backhole" cluster based on existing configuration
    2. Envoy makes a request to ext_authz endpoint. The service providing this endpoint blocks the auth request while it figures out where it should be routed.
    3. The service pushes cluster and route information to Envoy via xDS.
    4. Service responds to the blocked auth request with DeniedHttpResponse that contains 302 redirect to the same URL.
    5. Client receives the 302 redirect response and re-issues the request.
    6. When envoy gets this second request, routing rule and cluster definition are already configured so it routes the request to the correct destination.