I'm trying to setup an example ASP.NET project that uses envoy to route between the secure and non-secure versions of the app. The application works fine in Docker on both http and https, but when I try to route to it through envoy I get 'no healthy upstream' on the http site, and ERR_EMPTY_RESPONSE on the https site.
My envoy.yaml:
static_resources:
listeners:
- name: listener_http
address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/admin"
direct_response:
status: 403
body:
inline_string: "Forbidden, yo"
- match:
prefix: "/"
route:
cluster: exampleagg-http
- name: listener_https
address:
socket_address:
address: 0.0.0.0
port_value: 443
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/admin"
direct_response:
status: 403
body:
inline_string: "Forbidden, yo"
- match:
prefix: "/"
route:
cluster: exampleagg-https
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain:
filename: /etc/https/aspnetapp.crt
private_key:
filename: /etc/https/aspnetapp.key
clusters:
- name: exampleagg-http
type: LOGICAL_DNS
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: exampleagg-http
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: http://example-api/WeatherForecast
port_value: 80
- name: exampleagg-https
type: LOGICAL_DNS
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: exampleagg-https
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: https://example-api/WeatherForecast
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain:
filename: /etc/https/aspnetapp.crt
private_key:
filename: /etc/https/aspnetapp.key
My docker-compose.yaml:
networks:
envoy:
name: envoy
services:
api-gateway:
image: envoyproxy/envoy:v1.23-latest
container_name: api-gateway
volumes:
- ./ApiGateways/Envoy/config:/etc/envoy
- ${USERPROFILE}/.aspnet/https:/etc/https/
networks:
- envoy
ports:
- "8080:80"
- "8081:443"
depends_on:
- example-api
example-api:
image: ${REGISTRY:-hexsorcerer}/example-proxy-envoy:${PLATFORM:-linux}-${TAG:-latest}
container_name: example-api
volumes:
- ${USERPROFILE}/.aspnet/https:/https/
environment:
ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "https://+;http://+"
ASPNETCORE_HTTPS_PORT: 443
ASPNETCORE_Kestrel__Certificates__Default__Password: "password"
ASPNETCORE_Kestrel__Certificates__Default__Path: /https/aspnetapp.pfx
networks:
- envoy
expose:
- "80"
- "443"
ports:
- "5000:80"
- "5001:443"
build:
context: .
dockerfile: Services/Example/Example.API/Dockerfile
The Dockerfile for the example application:
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
# It's important to keep lines from here down to "COPY . ." identical in all Dockerfiles
# to take advantage of Docker's build cache, to speed up local container builds
COPY "ExampleEnvoyProxy.sln" "ExampleEnvoyProxy.sln"
COPY "Services/Example/Example.API/Example.API.csproj" "Services/Example/Example.API/Example.API.csproj"
#RUN dotnet restore "ExampleEnvoyProxy.sln"
COPY . .
WORKDIR /src/Services/Example/Example.API
RUN dotnet publish -c Release -o /app
EXPOSE 80 443
FROM build AS publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "Example.API.dll"]
I've been hacking away at this for days and made some good progress, but just can't quite get there. Any help would be greatly appreciated.
After not getting any responses for a couple weeks, I was forced to keep hacking around on this and I finally figured it out.
I think the biggest problem with what I was doing was a misunderstanding about how proxy servers route traffic. I was attempting to route the incoming path '/' to a backed service path of '/WeatherForecast', but that's not how it works. Your incoming endpoint will be "passed along" to the backend service, it's simply the cluster that you choose to pass it to. This is probably clear to veterans but as someone who doesn't work on these often it wasn't obvious to me at first.
There was also an issue of certificates, and I ended up generating one for the app and one for envoy, each of which were different formats, that also had to be trusted on my machine. This took a little extra effort for the envoy cert, but it worked much better than trying to use a single cert for both.
I documented what I learned with some instructions and a fully working example here.