Search code examples
elasticsearchkibanafilebeat

Connecting filebeat to elasticsearch

I have been facing this problem throughout the day and I can't understand what I am doing wrong. I am a beginner in this and I followed a tutorial on how to get a complete setup between Filebeat, elasticsearch and kibana. Filebeat keeps on failing to connect to elasticsearch from the logs. Below is my code

filebeat.inputs:
  - type: log
  enabled: true
  paths:    
    - C:\ProgramData\Elastic\Elasticsearch\logs\*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1


setup.kibana:

output.elasticsearch:
  hosts: ["localhost:9200"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Here is the log

2019-05-22T02:28:02.352+0200    ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: This Beat requires the default distribution of Elasticsearch. Please upgrade to the default distribution of Elasticsearch from elastic.co, or downgrade to the oss-only distribution of beats
2019-05-22T02:28:02.352+0200    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 62 reconnect attempt(s)
2019-05-22T02:28:02.355+0200    INFO    elasticsearch/client.go:734 Attempting to connect to Elasticsearch version 5.5.0
2019-05-22T02:28:15.560+0200    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":3781,"time":{"ms":62}},"total":{"ticks":6640,"time":{"ms":94},"value":6640},"user":{"ticks":2859,"time":{"ms":32}}},"handles":{"open":303},"info":{"ephemeral_id":"09bb9e79-0c2c-40fd-8a89-5098d60f3374","uptime":{"ms":2521080}},"memstats":{"gc_next":4259632,"memory_alloc":2907056,"memory_total":24455264,"rss":-8192}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":673},"write":{"bytes":260}},"pipeline":{"clients":1,"events":{"active":28,"retry":28}}},"registrar":{"states":{"current":5}}}}}
Solution

  • The error message is pretty clear

    Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: This Beat requires the default distribution of Elasticsearch. Please upgrade to the default distribution of Elasticsearch from elastic.co, or downgrade to the oss-only distribution of beats

    It seems you have a mismatch between your Filebeat version and the Elasticsearch version. You have installed filebeat-oss and you're trying to interact with a licensed Elasticsearch.

    So, in theory, you have two options:

    1. You can install the licensed Filebeat and keep your Elasticsearch as is
    2. You can downgrade to elasticsearch-oss and keep your Filebeat as is

    However, the way I see it, since you're using Elasticsearch 5.5.0 (old version), your only option would be to install Filebeat 5.6.16