Search code examples
kubernetesistiocert-manager

Waiting for HTTP-01 challenge propagation: failed to perform self check GET request - ISTIO

I get this error after waiting for a while ~1 min

Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://jenkins.xyz.in/.well-known/acme-challenge/AoV9UtBq1rwPLDXWjrq85G5Peg_Z6rLKSZyYL_Vfe4I': Get "http://jenkins.xyz.in/.well-known/acme-challenge/AoV9UtBq1rwPLDXWjrq85G5Peg_Z6rLKSZyYL_Vfe4I": dial tcp 103.66.96.201:80: connect: connection timed out

I am able to access this url in the browser from anywhere (internet)

curl -v http://jenkins.xyz.in/.well-known/acme-challenge/AoV9UtBq1rwPLDXWjrq85G5Peg_Z6rLKSZyYL_Vfe4I
*   Trying 103.66.96.201:80...
* Connected to jenkins.xyz.in (103.66.96.201) port 80 (#0)
> GET /.well-known/acme-challenge/AoV9UtBq1rwPLDXWjrq85G5Peg_Z6rLKSZyYL_Vfe4I HTTP/1.1
> Host: jenkins.xyz.in
> User-Agent: curl/7.71.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< cache-control: no-cache, no-store, must-revalidate
< date: Wed, 13 Jan 2021 08:54:23 GMT
< content-length: 87
< content-type: text/plain; charset=utf-8
< x-envoy-upstream-service-time: 1
< server: istio-envoy
< 
* Connection #0 to host jenkins.xyz.in left intact
AoV9UtBq1rwPLDXWjrq85G5Peg_Z6rLKSZyYL_VfT4I.EZvkP5Fpi6EYc_-tWTQgvaQxrrbSr2MEJkuXJaywatk

my setup is:

1. Istio Ingress load balancer running on node (192.168.14.118)
2. I am pointing my external IP and domain jenkins.xyz.in 
to 192.168.14.118 through an another load balancer

request -> public IP -> load balancer -> 192.168.14.118

From outside it works fine. but when I try this from node itself / from pod inside cluster I get :

$ curl -v http://jenkins.xyz.in/
* About to connect() to jenkins.xyz.in port 80 (#0)
*   Trying 103.66.96.201...

I have read somewhere about hairpinning

Since my kubernetes node IP and the istio ingress loadbalacer external IPs are same, request might be looping.

EXTRA: I am running k8s on bare metal

is there any solution to get around this?

Solution

  • I found a work around.

    As my node was not able to access the URL (loop), I added another node to cluster and set Cert-Manager pods affinity to new node.

    Cert-Manager was able to access the URL from new node. Although not a good solution, but worked for me.