Logging: Kibana integration with FluentD and Elasticsearch - how to integrate (Kubernetes) in a test environment?

Question

In a Kubernetes Docker Desktop TEST environment, I configured Fluentd, Elasticsearch and Kibana. To add an index pattern, I get this error message:

Security needs to be enabled in order to add Elastic Agent integrations

I have everything installed, how to just add an index pattern to the already existing Elasticsearch (FluentD) stream? This is just for a testing environment on my local Windows Docker Desktop Kubdernetes environment.

I tried many solutions as suggested in posts like this one.

Answer 1

Found it - finally! Security settings were not useful/needed in this test config.

I hope this will help you as well for discovering (Kubernetes) Docker logs in via FluentD > Elasticsearch > Kibana.

This works for a logging stack with FluentD > Elasticsearch v7 > Kibana v7.

• First check that the FluentD works. This excellent video on FluentD configs explains how you can check whether you configured FluentD rightly. In this article you can read about the same info.
• I used the configs for FluentD, Elasticsearch v7 and Kibana v7 from this github repo.
• You can then check whether Elasticsearch works using URL's like:
  • http://localhost:9200/ => gives status
  • http://localhost:9200/_cat/indices?v&pretty => gives indices. "Fluentd" should be in the list. If not, change the configuration of Elasticsearch.
  • http://localhost:9200/_search?q=counter => should give many logs on the sample application "counter".
• Go to Kibana (http://localhost:5601)
  • Menu > Integrations > containers > Elasticsearch Logs > Tab: windows > Button: Discover (or Check data)
  • You may have to enter "fluentd k8s" in an index.

Et voilà - you have the logs!

An example: just type the following in the search box: kubernetes.namespace_name=default and kubernetes.container_name=count