I have a number of services in a k8s cluster with Istio. I want the services to internally communicate with automatic mTLS and externally using a web-browser certificate from Let's Encrypt.
To accmplish the former, I have a peer authentication is the istio-system namespace:
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: peer-authentication
namespace: istio-system
spec:
mtls:
mode: STRICT
This is working fine for internal communication (my service pods are installed with label sidecar.istio.io/inject: "true").
I have configured an ingress gateway and a gateway
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
name: gateway
namespace: istio-ingress
spec:
selector:
istio: gateway
servers:
- port:
name: http
number: 80
protocol: HTTP
hosts:
- "*.customer.ocs.nu"
tls:
httpsRedirect: true
- port:
name: https
number: 443
protocol: HTTPS
hosts:
- "*.customer.ocs.nu"
tls:
credentialName: "istio-ingress/star-customer-ocs-nu-crt"
mode: SIMPLE
I have multiple applications I wish to expose; currently, I have this one:
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: application
namespace: customer-application
spec:
gateways:
- istio-ingress/gateway
- mesh
hosts:
- application.customer.ocs.nu
http:
- match:
- uri:
prefix: /
route:
- destination:
host: application.customer-application.svc.cluster.local
port:
number: 8000
(I've removed some irrelevant annotations + changed the name of the application + namespace, so typos in them are irrelevant)
Problem is now, I can connect on http:
# curl -kv http://application.customer.ocs.nu/
* Host application.customer.ocs.nu:80 was resolved.
* IPv6: (none)
* IPv4: IP
* Trying IP:80...
* Connected to application.customer.ocs.nu (IP) port 80
> GET / HTTP/1.1
> Host: application.customer.ocs.nu
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 301 Moved Permanently
< location: https://application.customer.ocs.nu/
< date: Fri, 31 Jan 2025 11:39:09 GMT
< server: istio-envoy
< content-length: 0
In the log, I see
[2025-01-31T11:39:09.561Z] "GET / HTTP/1.1" 301 - direct_response - "-" 0 0 0 - "10.244.0.165" "curl/8.7.1" "b05ac233-eea7-46d6-9fee-e9f9c9cf8bb8" "application.customer.ocs.nu" "-" - - 10.244.0.200:80 10.244.0.165:22533 - -
However, connecting via TLS, I get
# curl -kv https://application.customer.ocs.nu/
* Host application.customer.ocs.nu:443 was resolved.
* IPv6: (none)
* IPv4: IP
* Trying IP:443...
* Connected to application.customer.ocs.nu (IP) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to application.customer.ocs.nu:443
* Closing connection
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to application.customer.ocs.nu:443
and get nothing in the log. Trying to do HTTP on the HTTPs port, I get
# curl -kv http://application.customer.ocs.nu:443/
* Host application.customer.ocs.nu:443 was resolved.
* IPv6: (none)
* IPv4: IP
* Trying IP:443...
* Connected to application.customer.ocs.nu (IP) port 443
> GET / HTTP/1.1
> Host: application.customer.ocs.nu:443
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server
but this at least shows up in the log
[2025-01-31T11:43:01.227Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.244.0.200:443 10.244.0.165:44729 - -
If I reconfigure the gateway + virtualservice to use HTTP, everything works as expected. All Bing results suggest setting up redirection from HTTP to HTTPS, but I already have this.
istioctl analyze -A lists nothing significant (some services outside the mesh with illegal names + some namespaces without injection annotations), whereas I get
# istioctl pc secret istio-gateway-76676d4954-l5498.istio-ingress
RESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
kubernetes://istio-ingress/star-customer-ocs-nu-crt WARMING false
default Cert Chain ACTIVE true 12c998930e47b4c9df3f5ae259fb1a92 2025-02-01T03:04:23Z 2025-01-31T03:02:23Z
ROOTCA CA ACTIVE true c6b587095c06abdabc53c84b1af924d3 2035-01-18T12:59:47Z 2025-01-20T12:59:47Z
The certificate is provisioned by Let's Encrypt using certbot with DNS authentication and is perfectly valid. I assume the issue is that Istio is using its own CA for trust and does not trust my public certificate.
# kubectl get certificate -n istio-ingress
NAME READY SECRET AGE
star-customer-ocs-nu True star-customer-ocs-nu-crt 23h
# kubectl get certificaterequest -n istio-ingress
NAME APPROVED DENIED READY ISSUER REQUESTER AGE
star-customer-ocs-nu-1 True True letsencrypt-prod system:serviceaccount:default:cert-manager 23h
Does anybody have an idea to work around this? I'd prefer to use a Let's Encrypt certificate publicly (I don't want to issue certificated manually) without mTLS, and I'd prefer to use automatic internal mTLS.
E: Changing the tls section of the gateway to not include the namespace (while correct)
tls:
credentialName: star-customer-ocs-nu-crt
mode: SIMPLE
At least shows the certificate as valid (pod name change due to a restart to make sure it picks it up):
# istioctl pc secret istio-gateway-76676d4954-8hhjl.istio-ingress
RESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
default Cert Chain ACTIVE true 8101cda2556b2fd7c31872f9d013d72f 2025-02-01T12:31:02Z 2025-01-31T12:29:02Z
kubernetes://star-customer-ocs-nu-crt Cert Chain ACTIVE true 4c8a2f7ccab5ff0c7aa61dd2a46aa9bef0b 2025-04-30T11:56:26Z 2025-01-30T11:56:27Z
ROOTCA CA ACTIVE true c6b587095c06abdabc53c84b1af924d3 2035-01-18T12:59:47Z 2025-01-20T12:59:47Z
It was caused by an incorrect DestinationRule I wasn't thinking of:
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
labels:
app: application
name: application
namespace: application-customer
spec:
host: application
subsets:
- labels:
app: application
name: default
(the host should be application.customer.ocs.nu, not just application).