We have set up Istio, and we are using ISTIO ingress gateway for inbound traffic. We have set up TLS for TCP port. Sample code can be found here.
We also enabled logs by following this ISTIO guide
We tested the TLS connection using openssl and it works fine.
However, when we try to connect from an application, the TLS negotiation fails. I have provided more details with wireshark here
We would like to get logs from ISTIO on the TLS negotiation ... and find why it fails.
Istio Gateway YAML
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: dremio-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
tls:
httpsRedirect: true
hosts:
- testdomain.net
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: testdomain-credentials
hosts:
- testdomain.net
- port:
number: 31020
name: odbc-dremio-tls
protocol: tls
tls:
mode: SIMPLE
minProtocolVersion: TLSV1_0
maxProtocolVersion: TLSV1_3
credentialName: testdomain-credentials
hosts:
- testdomain.net
Virtual Service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: dremio
spec:
hosts:
- testdomain.net
gateways:
- dremio-gateway
http:
- match:
- port: 443
- port: 80
route:
- destination:
host: dremio-client
port:
number: 9047
tcp:
- match:
- port: 31020
route:
- destination:
host: dremio-client
port:
number: 31010
Partial Config Dump
{
"name": "0.0.0.0_31020",
"active_state": {
"version_info": "2020-07-21T12:11:49Z/9",
"listener": {
"@type": "type.googleapis.com/envoy.api.v2.Listener",
"name": "0.0.0.0_31020",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 31020
}
},
"filter_chains": [
{
"filter_chain_match": {
"server_names": [
"testdomain.net"
]
},
"filters": [
{
"name": "istio.stats",
"typed_config": {
"@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
"type_url": "type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm",
"value": {
"config": {
"root_id": "stats_outbound",
"vm_config": {
"vm_id": "tcp_stats_outbound",
"runtime": "envoy.wasm.runtime.null",
"code": {
"local": {
"inline_string": "envoy.wasm.stats"
}
}
},
"configuration": "{\n \"debug\": \"false\",\n \"stat_prefix\": \"istio\"\n}\n"
}
}
}
},
{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"stat_prefix": "outbound|31010||dremio-client.dremio.svc.cluster.local",
"cluster": "outbound|31010||dremio-client.dremio.svc.cluster.local",
"access_log": [
{
"name": "envoy.file_access_log",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
"path": "/dev/stdout",
"format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
}
}
]
}
}
],
"transport_socket": {
"name": "envoy.transport_sockets.tls",
"typed_config": {
"@type": "type.googleapis.com/envoy.api.v2.auth.DownstreamTlsContext",
"common_tls_context": {
"tls_params": {
"tls_minimum_protocol_version": "TLSv1_0",
"tls_maximum_protocol_version": "TLSv1_3"
},
"alpn_protocols": [
"h2",
"http/1.1"
],
"tls_certificate_sds_secret_configs": [
{
"name": "testdomain-credentials",
"sds_config": {
"api_config_source": {
"api_type": "GRPC",
"grpc_services": [
{
"google_grpc": {
"target_uri": "unix:/var/run/ingress_gateway/sds",
"stat_prefix": "sdsstat"
}
}
]
}
}
}
]
},
"require_client_certificate": false
}
}
}
],
"listener_filters": [
{
"name": "envoy.listener.tls_inspector",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.listener.tls_inspector.v2.TlsInspector"
}
}
],
"traffic_direction": "OUTBOUND"
},
"last_updated": "2020-07-21T12:11:50.303Z"
}
}
By enabling tracing on Envoy conn_handler, We can see the following message:
closing connection: no matching filter chain found
After getting the message of no matching filter chain, I found the filter chain for the port 31020 with the domain that I have provided in my Gateway config. It looks like while connecting my application(ODBC), the host was not being provided.
The solution is simply to replace the host domain by '*'
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: dremio-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
tls:
httpsRedirect: true
hosts:
- testdomain.net
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: testdomain-credentials
hosts:
- testdomain.net
- port:
number: 31020
name: odbc-dremio-tls
protocol: tls
tls:
mode: SIMPLE
minProtocolVersion: TLSV1_0
maxProtocolVersion: TLSV1_3
credentialName: testdomain-credentials
hosts:
- '*'