Cert-manager fails to issue gateway api certificate: "propagation check failed" err="wrong status code '404', expected '200'"

Question

I'm trying to issue a TLS certificate using Cert-Manager and Cilium Gateway (Gateway API). Everything works great when I use a self-signed ClusterIssuer, but fails when I move to the let's encrypt.

Cert-manager creates the http solvers, but I can see an error in the challenger:

Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'

Here's my issuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    email: [email protected]
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-private-key
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: gateway
                namespace: default
                kind: Gateway

Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  gatewayClassName: cilium
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All

    - name: https-landing
      protocol: HTTPS
      port: 443
      hostname: www.mypage.com
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            group: ""
            name: landing-tls

HTTPRoute:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: https-landing
  namespace: default
spec:
  parentRefs:
    - name: gateway
      sectionName: https-landing
  hostnames:
    - www.mypage.com
  rules:
    - backendRefs:
        - name: landing
          port: 8000

Do you have any idea what can be wrong?

Answer 1

Try below troubleshooting steps, which may help to resolve your issue:

• Based on the error, the issue seems to be that IngressRoute which is used in Cilium does not work with cert manager and is unable to reach the HTTP-01 challenge endpoint. Ensure that the HTTP route and gateway configurations allow traffic on the challenge endpoint on port 80.
• You will need to verify there is no misconfiguration in your HTTP route that prevents traffic from reaching certmanger Http solver. Follow this official cilium HTTPS Example about how to configure cilium Gateway to work with cert-manager.
• There might be delay in your DNS propagation for your domain.Verify that your domain’s DNS settings are correcting pointing to the external IP of your gateway.
• Check your Cilium logs of the certmanager to see more information about the error.

Refer to similar Github issue1 and issue2 for more details.