I am trying to use externalname together with nodeport service for Nginx-Controller but I am getting 502 Bad Gateway

Environment:

I have:

1- NGINX Ingress controller version: 1.15.9, image: 0.23.0

2- Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: Virtual Machines on KVM

OS (e.g. from /etc/os-release):

NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel (e.g. uname -a):

Linux node01 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Install tools: kubeadm

More details:

CNI : WEAVE

Setup:

2 Resilient HA Proxy, 3 Masters, 2 infra, and worker nodes.
I am exposing all the services as node ports, where the HA-Proxy re-assign them to a public virtual IP.
Dedicated project hosted on the infra node carrying the monitoring and logging tools (Grafana, Prometheus, EFK, etc)
Backend NFS storage as persistent storage

What happened: I want to be able to use external Name rather than node ports, so instead of accessing grafana for instance via vip + 3000 I want to access it via http://grafana.wild-card-dns-zone

Deployment

I have created a new namespace called ingress
I deployed it as follow:


apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: **2**
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
      name: nginx-ingress
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        node-role.kubernetes.io/infra: infra
      terminationGracePeriodSeconds: 60
      containers:
      - name: nginx-ingress-controller
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        args:
          - /nginx-ingress-controller
          - --default-backend-service=ingress/ingress-controller-nginx-ingress-default-backend
          - --configmap=$(POD_NAMESPACE)/nginx-configuration
          - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
          - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          - --publish-service=$(POD_NAMESPACE)/ingress-nginx
          - --annotations-prefix=nginx.ingress.kubernetes.io
          - --v3
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            drop:
              - ALL
            add:
              - NET_BIND_SERVICE
          # www-data -> 33
          runAsUser: 33
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  generation: 1
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.3.1
    component: default-backend
  name: ingress-controller-nginx-ingress-default-backend
  namespace: ingress
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx-ingress
      component: default-backend
      release: ingress-controller
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-ingress
        component: default-backend
        release: ingress-controller
    spec:
      nodeSelector:
        node-role.kubernetes.io/infra: infra
      containers:
      - image: k8s.gcr.io/defaultbackend:1.4
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: nginx-ingress-default-backend
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 60
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    name: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.3.1
    component: default-backend
  name: ingress-controller-nginx-ingress-default-backend
  namespace: ingress
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: nginx-ingress
    component: default-backend
    release: ingress-controller
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrolebinding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-rolebinding
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress

INGRESS SETUP:

services


# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-03-25T16:03:01Z"
  labels:
    app: jaeger
    app.kubernetes.io/component: query
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  name: jeager-query
  namespace: monitoring-logging
  resourceVersion: "3055947"
  selfLink: /api/v1/namespaces/monitoring-logging/services/jeager-query
  uid: 778550f0-4f17-11e9-9078-001a4a16021e
spec:
  externalName: jaeger.example.com
  ports:
  - port: 16686
    protocol: TCP
    targetPort: 16686
  selector:
    app: jaeger
    app.kubernetes.io/component: query
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-03-25T15:40:30Z"
  labels:
    app: grafana
    chart: grafana-2.2.4
    heritage: Tiller
    release: grafana
  name: grafana
  namespace: monitoring-logging
  resourceVersion: "3053698"
  selfLink: /api/v1/namespaces/monitoring-logging/services/grafana
  uid: 51b9d878-4f14-11e9-9078-001a4a16021e
spec:
  externalName: grafana.example.com
  ports:
  - name: http
    port: 3000
    protocol: TCP
    targetPort: 3000
  selector:
    app: grafana
    release: grafana
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

INGRESS

Ingress 1

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/service-upstream: "true"
  creationTimestamp: "2019-03-25T21:13:56Z"
  generation: 1
  labels:
    app: jaeger
    app.kubernetes.io/component: query-ingress
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  name: jaeger-query
  namespace: monitoring-logging
  resourceVersion: "3111683"
  selfLink: /apis/extensions/v1beta1/namespaces/monitoring-logging/ingresses/jaeger-query
  uid: e6347f6b-4f42-11e9-9e8e-001a4a16021c
spec:
  rules:
  - host: jaeger.example.com
    http:
      paths:
      - backend:
          serviceName: jeager-query
          servicePort: 16686
status:
  loadBalancer: {}

Ingress 2

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"labels":{"app":"grafana"},"name":"grafana","namespace":"monitoring-logging"},"spec":{"rules":[{"host":"grafana.example.com","http":{"paths":[{"backend":{"serviceName":"grafana","servicePort":3000}}]}}]}}
  creationTimestamp: "2019-03-25T17:52:40Z"
  generation: 1
  labels:
    app: grafana
  name: grafana
  namespace: monitoring-logging
  resourceVersion: "3071719"
  selfLink: /apis/extensions/v1beta1/namespaces/monitoring-logging/ingresses/grafana
  uid: c89d7f34-4f26-11e9-8c10-001a4a16021d
spec:
  rules:
  - host: grafana.example.com
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 3000
status:
  loadBalancer: {}

EndPoints

Endpoint 1

apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: "2019-03-25T15:40:30Z"
  labels:
    app: grafana
    chart: grafana-2.2.4
    heritage: Tiller
    release: grafana
  name: grafana
  namespace: monitoring-logging
  resourceVersion: "3050562"
  selfLink: /api/v1/namespaces/monitoring-logging/endpoints/grafana
  uid: 51bb1f9c-4f14-11e9-9e8e-001a4a16021c
subsets:
- addresses:
  - ip: 10.42.0.15
    nodeName: kuinfra01.example.com
    targetRef:
      kind: Pod
      name: grafana-b44b4f867-bcq2x
      namespace: monitoring-logging
      resourceVersion: "1386975"
      uid: 433e3d21-4827-11e9-9e8e-001a4a16021c
  ports:
  - name: http
    port: 3000
    protocol: TCP

Endpoint 2

apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: "2019-03-25T16:03:01Z"
  labels:
    app: jaeger
    app.kubernetes.io/component: service-query
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  name: jeager-query
  namespace: monitoring-logging
  resourceVersion: "3114702"
  selfLink: /api/v1/namespaces/monitoring-logging/endpoints/jeager-query
  uid: 7786d833-4f17-11e9-9e8e-001a4a16021c
subsets:
- addresses:
  - ip: 10.35.0.3
    nodeName: kunode02.example.com
    targetRef:
      kind: Pod
      name: jeager-query-7d9775d8f7-2hwdn
      namespace: monitoring-logging
      resourceVersion: "3114693"
      uid: fdac9771-4f49-11e9-9e8e-001a4a16021c
  ports:
  - name: query
    port: 16686
    protocol: TCP

I am able to curl the endpoints from inside the ingress-controller pod:

# kubectl exec -it nginx-ingress-controller-5dd67f88cc-z2g8s  -n ingress -- /bin/bash
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ curl -k https://localhost
<a href="/login">Found</a>.

www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ curl http://localhost
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.15.9</center>
</body>
</html>
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ exit

But from out side when I am trying to reach jaeger.example.com or grafana.example.com I am getting 502 bad gatway and the following error log:

10.39.0.0 - [10.39.0.0] - - [25/Mar/2019:16:40:32 +0000] "GET /search HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 514 0.001 [monitoring-logging-jeager-query-16686] vip:16686, vip:16686, vip:16686 0, 0, 0 0.001, 0.000, 0.000 502, 502, 502 b7c813286fccf27fffa03eb6564edfd1
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
10.39.0.0 - [10.39.0.0] - - [25/Mar/2019:16:40:32 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://jeager.example.com/search" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 494 0.001 [monitoring-logging-jeager-query-16686] vip:16686, vip:16686, vip:16686 0, 0, 0 0.000, 0.001, 0.000 502, 502, 502 9e582912614e67dfee6be1f679de5933
I0325 16:40:32.497868       8 socket.go:225] skiping metric for host jeager.example.com that is not being served
I0325 16:40:32.497886       8 socket.go:225] skiping metric for host jeager.example.com that is not being served

Solution

First thanks for cookiedough for the clue to help regarding the service issue, but later I faced an issue to create service using external name but I found my mistake thanks for "Long" user in the slack, the mistake is that I was using service of type ExternalName and it should be type cluster IP here are the steps to solve the problems (Remark https issue is a separate problem): 1- Create wild character DNS zone pointing the public IP 1- For new service just create it of type ClusterIP 2- In the namespace for the service create an ingress using the following example (yaml):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: grafana-namespace
spec:
  rules:   
    host: grafana.example.com
    http:
      paths:
        backend:
          serviceName: grafana
          servicePort: 3000

3- kubectl -f apply -f grafana-ingress.yaml Now you can reach your grafana on http://grafana.example,com