state_replicaset/state_replicaset.go98 error making http request: Get kube-state-metrics:8080/metrics: lookup kube-state-metrics on IP:53 no such host

Question

We are trying to start metricbeat on typhoon kubernetes cluster. But after startup its not able to get some pod specific events like restart etc because of the following

Corresponding metricbeat.yaml snippet

# State metrics from kube-state-metrics service:
    - module: kubernetes
      enabled: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_statefulset
        - state_pod
        - state_container
        - state_cronjob
        - state_resourcequota
        - state_service
        - state_persistentvolume
        - state_persistentvolumeclaim
        - state_storageclass
        # Uncomment this to get k8s events:
        #- event  period: 10s
      hosts: ["kube-state-metrics:8080"]

Error which we are facing

2020-07-01T10:31:02.486Z        ERROR   [kubernetes.state_statefulset]  state_statefulset/state_statefulset.go:97       error making http request: Get http://kube-state-metrics:8080/metrics: lookup kube-state-metrics on *.*.*.*:53: no such host
2020-07-01T10:31:02.611Z        WARN    [transport]     transport/tcp.go:52     DNS lookup failure "kube-state-metrics": lookup kube-state-metrics on *.*.*.*:53: no such host
2020-07-01T10:31:02.611Z        INFO    module/wrapper.go:259   Error fetching data for metricset kubernetes.state_node: error doing HTTP request to fetch 'state_node' Metricset data: error making http request: Get http://kube-state-metrics:8080/metrics: lookup kube-state-metrics on *.*.*.*:53: no such host
2020-07-01T10:31:03.313Z        ERROR   process_summary/process_summary.go:102  Unknown or unexpected state <P> for process with pid 19
2020-07-01T10:31:03.313Z        ERROR   process_summary/process_summary.go:102  Unknown or unexpected state <P> for process with pid 20

I can add some other info which is required for this.

Answer 1

Make sure you have the Kube-State-Metrics deployed in your cluster in the kube-system namespace to make this work. Metricbeat will not come with this by default.

Please refer this for detailed deployment instructions.