How to do pod load balancing with statefulset in k8s?

Question

From the document, it can be found there is a Stable Network ID feature to use for Pod NDS:

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id

I tried to do

service.yaml

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: logstash
  name: logstash
spec:
  selector:
    app: logstash
  ports:
    - name: "5044"
      port: 5044
      targetPort: 5044

statefulset.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: logstash
spec:
  serviceName: "logstash"
  selector:
    matchLabels:
      app: logstash
  updateStrategy:
    type: RollingUpdate
  replicas: 2
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:7.10.0
        resources:
          limits:
            memory: 2Gi
        ports:
          - containerPort: 5044
        volumeMounts:
          - name: config-volume
            mountPath: /usr/share/logstash/config
          - name: logstash-pipeline-volume
            mountPath: /usr/share/logstash/pipeline
        command: ["/bin/sh","-c"]
        args:
          - bin/logstash -f /usr/share/logstash/pipeline/logstash.conf;
      volumes:
        - name: config-volume
          configMap:
            name: configmap-logstash
            items:
              - key: logstash.yml
                path: logstash.yml
        - name: logstash-pipeline-volume
          configMap:
            name: configmap-logstash
            items:
              - key: logstash.conf
                path: logstash.conf

configmap.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-filebeat
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/lib/nginx/access.json

    output.logstash:
      hosts: ["logstash-0.logstash.default.svc.cluster.local:5044", "logstash-1.logstash.default.svc.cluster.local:5044"]
      loadbalance: true

Filebeat is deploying into a daemonset use this configuration.

It can't work. From the filebeat's log got:

2020-12-22T02:10:34.395Z    WARN    [transport] transport/tcp.go:52 DNS lookup failure "logstash-1.logstash.default.svc.cluster.local": lookup logstash-1.logstash.default.svc.cluster.local: no such host

If use this config, it can work:

    output.logstash:
      hosts: ["logstash.default.svc.cluster.local:5044"]

Why it caused DNS lookup failure issue when use Pod DNS format? Is there any more conditions to use this feature? Then how to do?

Answer 1

Also note from the StatefulSet documentation:

As mentioned in the limitations section, you are responsible for creating the Headless Service responsible for the network identity of the pods.

So you are resposible for creating Headless Services for the pods, e.g. logstash-0 and logstash-1.