Search code examples
kuberneteskubernetes-ingresstraefikk3straefik-ingress

Access application in cluster via node IP address connection refuse

(The final goal of what I am doing is to access my application via node IP address)

My cluster is using Traefik ingress controller. The Ingress rule is:

      kind: Ingress
      ...
      ...
      spec:
        rules:
          - host: my.app
            http:
              paths:
                - pathType: ImplementationSpecific
                  path: /
                  backend:
                    service:
                      name: my-app-svc
                      port:
                        number: 3000

There is also a LoadBalancer service deployed:

k get svc -n kube-system traefik
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
traefik   LoadBalancer   10.43.58.233   192.168.10.82   80:32003/TCP,443:30554/TCP   40d

As you can see the external IP is 192.168.10.82 for this LoadBalancer service.

On my laptop, I map that external IP with the hostname in /etc/hosts:

192.168.10.82 my.app

I can successfully access my deployed application via the external IP: http://my.app

Now, I would like to access it via IP address instead. This is what I try to do below: enter image description here

So, I have deployed a NodePort service for the controller:

      apiVersion: v1
      kind: Service
      metadata:
        name: traefik-node-port-svc
        namespace: kube-system
      spec:
        type: NodePort
        ports:
        - name: http
          port: 80
          targetPort: 80
          nodePort: 30182
          protocol: TCP
        - name: https
          port: 443
          targetPort: 443
          nodePort: 30183
          protocol: TCP
        selector:
          app: traefik

I can see this NodePort service is deployed:

NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
traefik-node-port-svc   NodePort   10.43.148.128   <none>        80:30182/TCP,443:30183/TCP   69m

As you can see above, the exposed port for HTTP traffic is 30182.

Then, I get the node (I have only one node):

k get node -o wide
NAME        STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE              KERNEL-VERSION     CONTAINER-RUNTIME
testk3s01   Ready    control-plane,master   40d   v1.21.5+k3s2   192.168.10.82   <none>        k3OS v0.21.5-k3s2r1   5.4.0-88-generic   containerd://1.4.11-k3s1

I can see above that the node's internal IP is 192.168.10.82.

I then try to access my application via <node internal IP>:<NodePort svc exposed port>

> curl 192.168.10.82:30182
curl: (7) Failed to connect to 192.168.10.82 port 30182 after 7 ms: Connection refused

But connection is refused, why is that? How can I access my application via IP address directly?

The NodePort service detail is like following:

> kubectl describe svc traefik-node-port-svc -n kube-system
Name:                     traefik-node-port-svc
Namespace:                kube-system
Labels:                   <none>
Annotations:              <none>
Selector:                 app=traefik
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.148.128
IPs:                      10.43.148.128
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  30182/TCP
Endpoints:                <none>
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  30183/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

==== Update based on @Sibtain's answer ====

I tried the option 2 of @Sibtain's answer to figure out the correct targetPort for my NodePort service.

I first try to find the targetPort from that LoadBalancer service:

> kubectl get svc traefik -o yaml -n kube-system

...
...
ports:
  - name: web
    nodePort: 32003
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 30554
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer

As you can see above, the targetPort which points to the application pod is 80 for http & 443 https respectively. I can also see the Selector section used by this service.

So, I updated my NodePort Service manifest to point to the same targetPort:

      apiVersion: v1
      kind: Service
      metadata:
        name: traefik-node-port-svc
        namespace: kube-system
      spec:
        type: NodePort
        ports:
        - name: web
          port: 80
          targetPort: web
          nodePort: 30182
          protocol: TCP
        - name: websecure
          port: 443
          targetPort: websecure
          nodePort: 30183
          protocol: TCP
        selector:
          app.kubernetes.io/name: traefik
          app.kubernetes.io/instance: traefik

I re-deployed this NodePort Service. But when check again, it still has <none> endpoint:

> kubectl describe svc traefik-node-port-svc -n kube-system
Name:                     traefik-node-port-svc
Namespace:                kube-system
Labels:                   <none>
Annotations:              <none>
Selector:                 app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik,app=traefik
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.148.128
IPs:                      10.43.148.128
Port:                     web  80/TCP
TargetPort:               web/TCP
NodePort:                 web  30182/TCP
Endpoints:                <none>
Port:                     websecure  443/TCP
TargetPort:               websecure/TCP
NodePort:                 websecure  30183/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Now all the services under the kube-system shows me this:

kubectl get svc -n kube-system
NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
kube-dns                ClusterIP      10.43.0.10      <none>          53/UDP,53/TCP,9153/TCP       41d
metrics-server          ClusterIP      10.43.111.166   <none>          443/TCP                      41d
traefik                 LoadBalancer   10.43.58.233    192.168.10.82   80:32003/TCP,443:30554/TCP   40d
traefik-node-port-svc   NodePort       10.43.148.128   <none>          80:30182/TCP,443:30183/TCP   19h

The curl result:

> curl 192.168.10.82:30182
curl: (7) Failed to connect to 192.168.10.82 port 30182 after 10 ms: Connection refused

> curl 192.168.10.82:80
404 page not found

> curl 192.168.10.82:30183
curl: (7) Failed to connect to 192.168.10.82 port 30183 after 860 ms: Connection refused

> curl 192.168.10.82:443
404 page not found
Solution

  • Your Service traefik-node-port-svc has no Endpoints attached to it (note the <none> in describe output)

    This means there are no pods that the service can redirect the request to. Most likely you have an incorrect TargetPort or the selector label (app: traefik) is wrong.

    To fix the targetPort you can opt any of the two options below:

    1. Run the kubectl describe pod <traefik-pod-here> and note down the port. That should be the targetPort in traefik-node-port-svc specification.
    2. Run kubectl describe svc traefik -n kube-system and note the targetPort from there as you have an already running LoadBalancer Service so it means that it is redirecting the request to the appropriate pods.

    If the issue is with the selector label, you can check the labels by running kubectl get pod <traefik-pod-here> --show-labels and add the appropriate labels.

    EDIT

    The easier way to create a Service and make sure you get the labels right is to use the declarative kubectl expose command, assuming that the traefik pod is running as a Deployment named traefik on port 9000, the command would look something like: (Note: the actual nodePort would be allocated automatically from the range 30000-32767)

    kubectl expose deployment traefik --name=traefik-node-port-svc --type=NodePort --port=80 --target-port=8000