Search code examples
elasticsearchelastic-stackfilebeat

Filebeat index is getting created but with 0 documents

I am trying to index my custom log file using filebeat. I am successfully running filebeat with pre-built modules like mysql, nginx etc. But when I actually try to use it with my application specific log file, index is created with 0 documents.

I could not find anywhere in the filebeats document if there are any specific steps need to be taken to ensure indexing takes place for the custom log files.

I did not get any error when I setup filebeats or run filebeats post setup.

Below is the filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Applications/MAMP/htdocs/247around-adminp-aws/application/logs/log-2020-12-21.log
  include_lines: ['^INFO', '^ERROR']
  fields:
    app_id: crm
filebeat.config.modules:
setup.template.settings:
  index.number_of_shards: 1
  path: ${path.config}/modules.d/*.yml
setup.kibana:
output.elasticsearch:
  hosts: ["localhost:9200"]
processors:

As can be seen, it is majorly default .yml file with very minor changes.

My custom log file log-2020-12-21.php is:

INFO  - 2020-12-21 15:10:26 --> index Logging details have been captured for employee. Details are : Array
INFO  - 2020-12-21 15:10:36 --> editpartner partner_id:1
INFO  - 2020-12-21 15:10:36 --> SELECT DISTINCT service_id, brand, active
ERROR - 2020-12-21 15:10:36 --> Query error: Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'boloaaka.collateral.id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
INFO  - 2020-12-21 15:10:36 -->  Database Error: A Database Error Occurred<br/>Array
ERROR - 2020-12-21 15:10:54 --> Query error: Expression #5 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'boloaaka.service_centres.district' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
INFO  - 2020-12-21 15:10:54 -->  Database Error: A Database Error Occurred<br/>Array
INFO  - 2020-12-21 23:53:21 --> Loginindex
INFO  - 2020-12-21 23:54:50 --> Loginindex
INFO  - 2020-12-21 23:55:42 --> Loginindex
INFO  - 2020-12-21 23:56:24 --> Loginindex

Index file is getting created with 0 documents: Filebeats index with 0 document

Log file showing logs for filebeats setup and filebeats running: https://pastebin.com/TK6uYXuq

Please help:

  1. Why there are no error messages if something is wrong because of which documents are not getting indexed? I should be getting some error if things are not right.
  2. How should I index my log file?
  3. Where should I add pattern for my log file like key-value pair which would help me in searching the documents for relevant values later on?

Thanks for your help.

Solution

  • In your filebeat configuration, are you sure you are referring to the exact file where your logs are stored? Your 'paths' in filebeat.yml is referring to a .log file extension while the custom log file you've pasted is log-2020-12-21.php Try changing your paths to match this .php extension instead. If filebeat correctly picks this file up, you could see something like the code below in your filebeat logs

    INFO log/harvester.go:287 Harvester started for file: /Applications/MAMP/htdocs/247around-adminp-aws/application/logs/log-2020-12-21.php