Search code examples
elasticsearchibm-cloudkibanaknative-servingknative

Knative logging: log stash - Kibana unable to fetch mapping

For Knative logging, following the instructions here - https://github.com/knative/docs/blob/master/serving/installing-logging-metrics-traces.md#elasticsearch-kibana-prometheus--grafana-setup, I tried to visualise the logs using Kibana UI (the visualization tool for Elasticsearch) but struck with the following error while configuring an index pattern — “ Unable to fetch mapping. Do you have indices matching the pattern?” logstash enter image description here

Is there any workaround or fix for this?

Update: Here's what I see when I make a cURL GET request as suggested in the commententer image description here

Solution

  • Here are some additional steps which I have to perform to make this completely work. Posting here so that this may help someone facing the same issue and looking for an answer

    Here are the steps, Run the below command to apply a patch to fix the fluentd-ds pods not showing issue

    kubectl apply -f https://raw.githubusercontent.com/gevou/knative-blueprint/master/knative-serving-release-0.2.2-patched.yaml

    Verify that each of your nodes have the beta.kubernetes.io/fluentd-ds-ready=true label:

    kubectl get nodes --selector beta.kubernetes.io/fluentd-ds-ready=true

    If you receive the No Resources Found response: Run the following command to ensure that the Fluentd DaemonSet runs on all your nodes:

    kubectl label nodes — all beta.kubernetes.io/fluentd-ds-ready=”true”

    Run the following command to ensure that the fluentd-ds daemonset is ready on at least one node:

    kubectl get daemonset fluentd-ds --namespace knative-monitoring

    enter image description here

    Wait for a while and run this command

    kubectl proxy

    Navigate to the Kibana UI. It might take a couple of minutes for the proxy to work.

    • Within the “Configure an index pattern” page, enter logstash-* to Index pattern and select @timestamp from Time Filter field name and click on Create button.

    • To create the second index, select Create Index Pattern button on top left of the page. Enter zipkin* to Index pattern and select timestamp_millis from Time Filter field name and click on Create button. enter image description here

    If the issue still exists, following the suggestions in the comments above should fix the error

    GET _cat/indices?v

    enter image description here

    Added the end-to-end findings here