Filebeat 7.10.1 add_docker_metadata adds only container.id

Question

I'm using filebeat 7.10.1 installed on host system (not docker container), running as service by root

according to https://www.elastic.co/guide/en/beats/filebeat/current/add-docker-metadata.html and https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-container.html

filebeat config, filebeat.yml:

filebeat.inputs:
- type: container
  enabled: true
  paths:
    - '/var/lib/docker/containers/*/*.log'
  processors:
    - add_docker_metadata: ~

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

setup.kibana:

output.logstash:
  hosts: ["<logstash_host>:5044"]

started container: docker run --rm -d -l my-label --label com.example.foo=bar -p 80:80 nginx

filebeat get logs and successfully send them to endpoint (in my case to logstash, which resend to elasticsearch), but generated json by filebeat contains only container.id without container.name, container.labels and container.image

it looks like (copy-paste from kibana):

{
  "_index": "logstash-2021.02.10",
  "_type": "_doc",
  "_id": "s4a4i3cB8j0XLXFVuyMm",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "ecs": {
      "version": "1.6.0"
    },
    "@timestamp": "2021-02-10T11:33:54.000Z",
    "host": {
      "name": "<some_host>"
    },
    "input": {
      "type": "container"
    },
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "log": {
      .....
    },
    "stream": "stdout",
    "container": {
      "id": "15facae2115ea57c9c99c13df815427669e21053791c7ddd4cd0c8caf1fbdf8c-json.log"
    },
    "agent": {
      "version": "7.10.1",
      "ephemeral_id": "adebf164-0b0d-450f-9a50-11138e519a27",
      "id": "0925282e-319e-49e0-952e-dc06ba2e0c43",
      "name": "<some_host>",
      "type": "filebeat",
      "hostname": "<some_host>"
    }
  },
  "fields": {
    "log.timestamp": [
      "2021-02-10T11:33:54.000Z"
    ],
    "@timestamp": [
      "2021-02-10T11:33:54.000Z"
    ]
  },
  "highlight": {
    "log.logger_name": [
      "@kibana-highlighted-field@gw_nginx@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1612956834000
  ]
}

what am I doing wrong? How to configure filebeat for send container.name, container.labels, container.image?

Answer 1

So after looking on filebeat-debug and paths on filesystem - issue closed

Reason: symlink /var/lib/docker -> /data/docker produces unexpected behavior

Solution:

filebeat.inputs:
  - type: container
    enabled: true
    paths:
      - '/data/docker/containers/*/*.log' #use realpath
    processors:
      - add_docker_metadata:
           match_source_index: 3 #subfolder for extract container id from path