Ingress rule, routing traffic to ingress controller service

Question

I need help to route traffic to one of Ingress controller's own service, metric service. I have deployed one Nginx Ingress Controller and enabled the metrics option in that which automatically created a service with name internal-controller-metrics after the name of controller internal-controller. The service has port 10254/TCP up to scrape the metrics which I can see if I make a curl request from within the cluster.

But I want to scrape those from the external Prometheus scraper . So, I wanted to route the request to this service, for which I wrote the below ingress rule . But the request curl http://dnsname.of.ingress.controller.com/metric/metrics, it responds with 404.

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /$2
      kubernetes.io/ingress.class: nginx-internal
    name: metrics-ingress
  spec:
    rules:
    - host: dnsname.of.ingress.controller.com
      http:
        paths:
        - backend:
            service:
              name: nginx-internal-controller-metrics
              port:
                number: 10254
          path: /metric(/|$)(.*)
          pathType: Prefix
kind: List

I'm not able to find anything in the controller logs why the request is not routing. Nginx controller version is nginx version: nginx/1.20.1 GKE node version 1.17.17.

Thanks

Answer 1

Here is what I did to get to the nginx-controller metrics from the outside:

Installed the nginx-controller with the following commands from the documentation:

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin \
  --user $(gcloud config get-value account)

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.0/deploy/static/provider/cloud/deploy.yaml

Then I changed the Service Manifest and the Deployment Manifest to expose the services as per the documentation (well, almost):

Service:

apiVersion: v1
kind: Service
metadata:
 annotations:
   prometheus.io/scrape: "true"
   prometheus.io/port: "10254"
..
spec:
  ports:
    - name: prometheus
      port: 10254
      targetPort: prometheus
      ..

Deployment (documentation changes the DaemonSet, which didn't exist for me):

..
ports:
  - name: prometheus
    containerPort: 10254
  ..

Then I check for the external IP of my controller:

k get svc -n ingress-nginx | grep 10254

ingress-nginx-controller             LoadBalancer   10.140.7.150   35.xxx.xx.xx   10254:32677/TCP,80:31999/TCP,443:30632/TCP   52m

Curling the external ip for metrics:

curl -s 35.xxx.xx.xx:10254/metrics | head -5

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.9442e-05
go_gc_duration_seconds{quantile="0.25"} 0.000166662
go_gc_duration_seconds{quantile="0.5"} 0.000188861

---

UPD: to make it work with Ingress, here is what I did:

I've created a Cloud DNS A record, pointing at my load balancer and then created an Ingress object from the following manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    kubernetes.io/ingress.class: nginx
  name: metrics-ingress
spec:
  rules:
  - host: <Cloud-DNS-A-record>
    http:
      paths:
      - backend:
          service:
            name: ingress-nginx-controller
            port:
              number: 10254
        path: /metric(/|$)(.*)
        pathType: Prefix

and applied it to my cluster

k apply -f ingress.yaml -n ingress-nginx

I then opened a this page in the web browsed to see the metrics: http://Cloud-DNS-A-Record/metric/metrics