Kubeflow+Istio+cert-manager http-01 challenge propagation: wrong status code '404', expected '200'

Question

I'm trying ssl secure kubeflow v1.6 deployment on AKS. The deployment uses Cert-manager v1.10.1 and Istio v1.16.0. The acme-solver service is using port 8089, but it can't be reached from outside or inside the cluster.

In simple words the request domain.com/.well-known/acme-challenge/token should get a response of 200 containing a validation token, but the problem is that it can't access the service running inside the cluster on port 8089 so it get error 404 instead. Cert-manager creates ingress by default to allow this request but it isn't working.

Here is the Ingress resource created by the acme-solver

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: istio
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
  generateName: cm-acme-http-solver-
  generation: 2
  labels:
    acme.cert-manager.io/http-domain: "700646687"
    acme.cert-manager.io/http-token: "903286350"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-m5km5
  namespace: istio-system
  ownerReferences:
  - apiVersion: acme.cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: example-domain-com-wq9x4-635963339-3112361122
    uid: a714c9e7-4152-4a67-849a-770fc8d049a2
  resourceVersion: "22275"
  uid: dd653c5b-8bcf-4275-a820-df1020005a5f
spec:
  rules:
  - host: example.domain.com
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-8bd7d
            port:
              number: 8089
        path: /.well-known/acme-challenge/6lN1QQNyGRfN9t9zVd8dxM9FWmWe13qDwF6owzXJ-Vk
        pathType: Exact
status:
  loadBalancer:
    ingress:
    - ip: 52.224.73.90

I tried to make a virtualService or another Ingress in the istio-system namespace but didn't work also. Here is the virtualService I tried to apply;

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-virtual-service
  namespace: istio-system
spec:
  hosts:
    - example.domain.com
  gateways:
    - istio-ingressgateway
  http:
    - match:
        - uri:
            prefix: "/.well-known/acme-challenge/"
      route:
        - destination:
            host: cm-acme-http-solver.istio-system.svc.cluster.local
            port:
              number: 8089

Answer 1

My problem appeared not to be related to istio or networking. I just needed to add the prefix .well-know/ to be skipped by the authentication service in kubeflow. It was preventing the domain.com/.well-known/acme-challenge/token url, so the solver pod couldn't reach it without authentication. So to overcome this, adding .well-known in the SKIP_AUTH_URI parameter found in path manifests-1.6.1/common/oidc-authservice/base/params.env solved the problem.