Search code examples
dockerkibana-7elasticsearch-7

Elasticsearch query slow response via kibana console

Server background : 3 node elasticsearch cluster + kibana + logstash running on docker environment. host server runs rhel7.7(2cpu, 8GB RAM + 200GB fileshare).

Versions :

  • elasticsearch 7.5.1
  • kibana 7.5.1
  • logstash 7.5.1
  • filebeat 7.5.1 (runs on separate server)
## Cluster health status
{
  "cluster_name" : "es-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 116,
  "active_shards" : 232,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

## Node status
172.20.1.3 60 91 13 0.98 1.30 1.45 dilm - elasticsearch2
172.20.1.4 57 91 13 0.98 1.30 1.45 dilm - elasticsearch3
172.20.1.2 61 91 14 0.98 1.30 1.45 dilm * elasticsearch

## Host server TOP output
top - 11:37:10 up 11 days, 22:30,  3 users,  load average: 0.74, 1.29, 1.47                                    
Tasks: 210 total,   1 running, 209 sleeping,   0 stopped,   0 zombie                                           
%Cpu(s):  4.4 us,  0.8 sy,  0.0 ni, 94.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st                                
KiB Mem :  7999840 total,   712736 free,  5842300 used,  1444804 buff/cache                                    
KiB Swap:  3071996 total,  2794496 free,   277500 used.  1669472 avail Mem                                     

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                    
48491 vtha837   20   0 4003724 713564  23880 S   5.0  8.9   1:57.53 java                                       
59023 vtha837   20   0 6796456   1.5g 172556 S   2.7 20.3  28:18.07 java                                       
59006 vtha837   20   0 6827100   1.5g 176840 S   1.7 19.9  30:03.59 java                                       
  760 vtha837   20   0 6877220   1.5g 180752 S   0.7 19.9  24:37.88 java                           
59610 vtha837   20   0 1663436 258152   7336 S   0.3  3.2  16:51.84 node         

## Kibana environment variables I used for kibana docker image
    environment:
      SERVER_NAME: "kibana"
      SERVER_PORT: 9548
      ELASTICSEARCH_PASSWORD: ${ES_PASSWORD}
      ELASTICSEARCH_HOSTS: "http://elasticsearch:9550"
      KIBANA_DEFAULTAPPID: "dashboard/Default"
      LOGGING_QUIET: "true"
      XPACK_SECURITY_ENCRYPTIONKEY: ${KIBANA_XPACK_SEC_KEY}
      XPACK_SECURITY_SESSIONTIMEOUT: 600000

Issue : A. When I run elasticsearch queries via kibana console at least took 20000 ms return output to the console. But if I run the same query directly(elasticsearch) via curl or postman or chrome it took only less than 200 ms to get the output B. even this happening when I loading kibana dashboad(not all the time), get following error message and not loading some graphs. but I can't see any exceptions or errors from console logs

Error in visualization

[esaggs] > Request to Elasticsearch failed: {"error":{}}

If I refresh the page, I can see all the graphs.

Chrome performance profile directly hitting elasticsearch query URL: http://testnode.mycompany.com.nz:9550/_cat/indices screenshot

Chrome performance profile via kibana dev console elasticsearch query Query : GET /_cat/indices screenshot

What I not understand is If I run same docker compose file in my laptop(windoes 10, 16GB, i7 2cpu, docker desktop running) i'm not facing any slowness either kibana dev console query or directly query elasticseach.

Anyone having this issue and appreciate let me know how to fix this?

Thanks in advance.

Solution

  • Issue is the docker service discovery. Due to some reason docker service discovery not happened. as soon as I change the elasticsearch host to IP get the real performance.

    In docker-compose Previous kibana configuration 

    ELASTICSEARCH_HOSTS: "http://elasticsearch:9550"

    New configuration

    ELASTICSEARCH_HOSTS: "http://172.20.1.2:9550"

    more details refer to the elasticsearch discuss page