im trying to setup a forward-proxy within eks, using an envoy sidecar-containers, which is supposed to conduct TLS origination; I.e. the envoy gets a http-request from the main-container, performs ssl-encryption and validates the ca using the file ca-certificates.crt. The envoy-configuration is based on the official documentation (https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/dynamic_forward_proxy_filter#config-http-filters-dynamic-forward-proxy).
The envoy mounts the ca-file ca-certificates.crt from aws secrets manager to /etc/cert-volume/ca-certificates.crt, which seems to work, since the file exists within the container.
The ca-certificates.crt file seems to be valid as well, since the following cmd, run from my local machine, gives a HTTP 302 Found (remote wget doesnt support --ca-certificate parameter):
wget --ca-certificate=ca-certificates.crt https://git.tech.rz.blub.de
--2021-04-26 11:03:16-- https://git.tech.rz.blub.de/
Loaded CA certificate 'ca-certificates.crt'
Resolving git.tech.rz.blub.de (git.tech.rz.blub.de)... 10.104.85.137, 10.104.84.192, 10.104.86.175
Connecting to git.tech.rz.blub.de (git.tech.rz.blub.de)|10.104.85.137|:443... connected.
HTTP request sent, awaiting response... 302 Found
When i open a shell on the envoy-Container and try to send a http-Request over the Proxy, i receive 503 Service Unavailable:
export http_proxy="localhost:10000"
wget git.tech.rz.blub.de
Connecting to localhost:10000 (127.0.0.1:10000)
wget: server returned error: HTTP/1.1 503 Service Unavailable
As the envoy-logs show, the request was processed and DNS resolution was successfull, however ca-validation failed with error TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
Logs can be found here: https://pastebin.com/pUZZqhSA
The complete envoy-configuration looks like this:
admin:
access_log_path: /dev/stdout
address:
socket_address: { address: 127.0.0.1, port_value: 9901, protocol: TCP }
static_resources:
secrets:
- name: server_cert
tls_certificate:
certificate_chain:
filename: /etc/cert-volume/server.crt
private_key:
filename: /etc/cert-volume/server.key
- name: validation_context
validation_context:
trusted_ca:
filename: /etc/cert-volume/ca-certificates.crt
listeners:
- name: listener_incoming
address:
socket_address: { address: 0.0.0.0, port_value: 8443 }
filter_chains:
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/config/" }
route: { cluster: local_config }
- match: { prefix: "/basic/" }
route: { cluster: local }
- match: { prefix: "/" }
route: { cluster: local }
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
log_format:
json_format:
protocol: "%PROTOCOL%"
response_code: "%RESPONSE_CODE%"
response_code_details: "%RESPONSE_CODE_DETAILS%"
connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%"
message: "%LOCAL_REPLY_BODY%"
response_flags: "%RESPONSE_FLAGS%"
local_reply_body: "%LOCAL_REPLY_BODY%"
http_filters:
- name: envoy.filters.http.router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificate_sds_secret_configs:
- name: server_cert
- name: listener_outgoing
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: AUTO
stat_prefix: http_to_https
route_config:
name: local_route
virtual_hosts:
- name: app
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: dynamic_forward_proxy_cluster
http_filters:
- name: envoy.filters.http.dynamic_forward_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: local
connect_timeout: 0.25s
type: STATIC
load_assignment:
cluster_name: local
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
- name: local_config
connect_timeout: 0.25s
type: STATIC
load_assignment:
cluster_name: local_config
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8081
- name: dynamic_forward_proxy_cluster
connect_timeout: 1s
lb_policy: CLUSTER_PROVIDED
cluster_type:
name: envoy.clusters.dynamic_forward_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
validation_context_sds_secret_config:
name: validation_context
The envoy-admin endpoint shows, that the correct certs are used:
wget localhost:9901/certs
Connecting to localhost:9901 (127.0.0.1:9901)
saving to 'certs'
certs 100% |*********************************************| 710 0:00:00 ETA
'certs' saved
/ # cat certs
{
"certificates": [
{
"ca_cert": [
{
"path": "/etc/cert-volume/ca-certificates.crt",
"serial_number": "612a55dd000000000002",
"subject_alt_names": [],
"days_until_expiration": "1305",
"valid_from": "2013-11-21T15:40:58Z",
"expiration_time": "2024-11-21T15:50:58Z"
}
],
"cert_chain": []
},
{
"ca_cert": [],
"cert_chain": [
{
"path": "/etc/cert-volume/server.crt",
"serial_number": "74af53b8463a1714",
"subject_alt_names": [
{
"dns": "*.test.comp.blub.de"
}
],
"days_until_expiration": "173",
"valid_from": "2019-10-18T07:10:38Z",
"expiration_time": "2021-10-17T07:10:38Z"
}
]
}
]
}
Does anyone have an idea, whats going wrong here?
Problem is solved. ca-certificates.crt was missing a required CA. I dont know why wget didnt throw an exception, however openssl did...