Search code examples
elasticsearchkuberneteskibanafluentd

Unable to access Kibana dashboard service in kubernetes cluser on Ubuntu


I am trying to access the Kibana dashboard while trying to set up fluentd-elasticsearch on premises. This is the link which I followed. I checked the logs of Kibana's pod. It shows the following error:

{"type":"log","@timestamp":"2018-09-19T21:45:42Z","tags":["warning","config","deprecation"],"pid":1,"message":"You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning."}
root@mTrainer3:/logging# kubectl logs kibana-logging-66d577d965-mbbg5 -n kube-system
{"type":"log","@timestamp":"2018-09-19T21:45:42Z","tags":["warning","config","deprecation"],"pid":1,"message":"You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning."}
{"type":"log","@timestamp":"2018-09-19T21:46:08Z","tags":["status","plugin:kibana@6.4.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

Could anybody suggest how I can resolve this issue?


Solution

  • After a discussion it was more clear what seems to be wrong.

    You are using a local cluster with no load balancer. You have to set either an ingress or use NodePort as the service type. I am going to describe the solution with NodePort. Two steps to take:

    1. Modify the kibana-deployment.yaml and remove the following under env:
    - name: SERVER_BASEPATH
      value: /api/v1/namespaces/kube-system/services/kibana-logging/proxy
    

    so that you kibana-deployment.yaml looks like:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kibana-logging
      namespace: kube-system
      labels:
        k8s-app: kibana-logging
        kubernetes.io/cluster-service: "true"
        addonmanager.kubernetes.io/mode: Reconcile
    spec:
      replicas: 1
      selector:
      matchLabels:
        k8s-app: kibana-logging
      template:
        metadata:
          labels:
            k8s-app: kibana-logging
          annotations:
            seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
        spec:
          containers:
          - name: kibana-logging
            image: docker.elastic.co/kibana/kibana-oss:6.3.2
            resources:
              # need more cpu upon initialization, therefore burstable class
              limits:
                cpu: 1000m
              requests:
                cpu: 100m
            env:
              - name: ELASTICSEARCH_URL
                value: http://elasticsearch-logging:9200
            ports:
            - containerPort: 5601
              name: ui
              protocol: TCP
    
    1. Modify kibana-service.yaml to set the service type to NodePort:
    apiVersion: v1
    kind: Service
    metadata:
      name: kibana-logging
      namespace: kube-system
      labels:
        k8s-app: kibana-logging
        kubernetes.io/cluster-service: "true"
        addonmanager.kubernetes.io/mode: Reconcile
        kubernetes.io/name: "Kibana"
    spec:
      type: NodePort
      ports:
      - port: 5601
        protocol: TCP
        targetPort: ui
        nodePort: 30601
      selector:
        k8s-app: kibana-logging
    

    Then execute

    kubectl apply -f kibana-deployment.yaml
    kubectl apply -f kibana-service.yaml
    

    Kibana should be accessible at http://<clusterip>:30601

    Background

    You will directly access http://clusterip:30601 without the given base path. So this must be removed, so that kibana is using / as base path. Otherwise it will try to append the base path /api/v1/[...] to your url. You can try it if you want to test it.

    This comment from an elastic search guy mentions, that you have to remove the base_path completely if you want to use /.

    Modifying the service to NodePort is neccessary as K8s does not publish ports by default. I just answered a similar issue on this question.

    Original answer (wrong)

    In the repo you were linking I can see that the kibana-deployment.yaml has to environment variables to set:

    env:
      - name: ELASTICSEARCH_URL
        value: http://elasticsearch-logging:9200
      - name: SERVER_BASEPATH
        value: /api/v1/namespaces/kube-system/services/kibana-logging/proxy
    

    Did you set them accordingly?

    Let's assume you have an ingress, loadbalancer or NodePort directly to the kibana instance so that you want to reach it directly with http://yourserver:9200/. Then the SERVER_BASEPATH is /