Search code examples
elasticsearchkibana

Kibana behaves different when run on different host than running on same host

The context: Installing and running Elasticsearch and Kibana.

I have followed the documentation documents to install using the .tar.gz files:

https://www.elastic.co/guide/en/elasticsearch/reference/current/targz.html https://www.elastic.co/guide/en/kibana/7.17/targz.html

For some reason, running Elasticsearch and Kibana on the same server behaves differently than running them on different servers. Elaboration:

Scenario 1 - same server

Installed Elasticsearch

  • Download elasticsearch.tar.gz, extracting it.
  • NOT configuring anything in config/elasticsearh.yaml
  • Running bin/elasticsearch to start Elasticsearch

On the same machine, installed kibana:

  • Download kibana.tar.gz, extracting it.
  • In config/kibana.yaml, only changed server.host to 0.0.0.0 so I can access it from browser.
  • Running bin/kibana to start Kibana

Result: Kibana runs perfectly:

LICENSE.txt  NOTICE.txt  README.txt  bin  config  data  logs  node  node_modules  package.json  plugins  src  x-pack
cloud_user@df993703751c:~/kibana-8.6.2$ ./bin/kibana
[2023-03-19T15:44:37.118+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2023-03-19T15:44:52.629+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
[2023-03-19T15:44:52.631+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
[2023-03-19T15:44:52.631+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
[2023-03-19T15:44:52.632+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
[2023-03-19T15:44:52.669+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
[2023-03-19T15:44:52.791+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
[2023-03-19T15:44:52.848+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2023-03-19T15:44:52.852+00:00][INFO ][preboot] "interactiveSetup" plugin is holding setup: Validating Elasticsearch connection configuration…
[2023-03-19T15:44:52.901+00:00][INFO ][root] Holding setup until preboot stage is completed.


i Kibana has not been configured.

Go to http://0.0.0.0:5601/?code=153896 to get started.

Scenario 2 - different servers

On server 1 - Installed Elasticsearch Just as I did in the first scenario

On server 2 - Installed Kibana

  1. Downloaded and extracted kibana.tar.gz
  2. In the kibana.yml, configured elasticsearch.hosts to the IP of the first server
  3. Ran Kibana

Result: Kibana doesn't trust self-signed certificate of Elasticsearch

[2023-03-19T16:06:30.499+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 22.04 OS. Automatically enabling Chromium sandbox.
[2023-03-19T16:06:30.716+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. self signed certificate in certificate chain
[2023-03-19T16:06:32.297+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /home/cloud_user/elasticsearch/administration/kibana-8.6.2/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell

First, my question is: in the first scenario, I didn't specify the path of the CA of Elasticsearch anywhere in the Kibana configuration, so why didn't Kibana threw an error there?

Anyway, handled this by copying the config/certs/http_ca.crt from Elasticsearch to the Kibana server, and then configured in config/kibana.yaml: elasticsearch.ssl.certificateAuthorities: [ "/path/to/ca_of_elasticsearch.crt" ]

nKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-03-19T16:10:57.296+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2023-03-19T16:10:57.363+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2023-03-19T16:10:58.312+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 22.04 OS. Automatically enabling Chromium sandbox.
[2023-03-19T16:10:58.452+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
[2023-03-19T16:10:59.359+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /home/cloud_user/elasticsearch/administration/kibana-8.6.2/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell

And now it throws Missing authentication credentials for REST request ... see above.

So the question is: Why when I run Kibana on a different host it suddenly wants credentials and verifies certificate, and don't do these when they run on the same host?

Solution

  • Use the following method:

    Scenario 2 - different servers

    1.Download and extract kibana.tar.gz

    2.to create an enrollment token for kibana elasticsearch-create-enrollment-token -s kibana

    3.sudo /usr/share/kibana/bin/kibana-setup --enrollment-token <enrollment-token>

    1. start kibana

    The second step will generate a token. Run the second command on the elasticsearch server. Use the generated enrollment-token and run the third command on the kibana server. This command will modify the kibana.yml and keystores, which kibana needs to connect to the elasticsearch.