How can I retrieve the tag from the syslog logs that are sent to Logstash?

I have set up my Docker daemon so that the logs of all my containers are forwarded to a Logstash application listening on port 5000, using the following configuration for daemon.json :

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "udp://localhost:5000",
    "syslog-format": "rfc3164",
    "tag": "{{.Name}}"
  },

  "hosts": [
    "tcp://0.0.0.0:2375",
    "unix:///var/run/docker.sock"
  ]
}

Since many different containers are creating logs at the same time, I would like to be able to filter the container names when I visualize their logs within my ELK stack. However, I'm not sure how I can retrieve, in Logstash, the "tag" that I have set as part of the "log-opts" in the Docker daemon configuration above.

What I tried is to simply retrieve it as a variable and forward it to a field in the Logstash configuration, but it just stores the text "%{tag}" as a string. Is it possible to retrieve the tag of the source container in the Logstash configuration?

logstash.conf :

input {
  udp {
    port => 5000
    type => syslog
  }
}
output {
  elasticsearch {
    hosts => ["elasticsearch"]
  }
}

filter {
  if [type] == "syslog" {
    if [message] =~ "^<\d+>\s*\w+\s+\d+\s\d+:\d+:\d+\s\S+\s\w+(\/\S+|)\[\d+\]:.*$" {
      grok {
        match => {
          "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:hostname} %{DATA:container_hash}(?:\[%{POSINT}\])?: %{GREEDYDATA:real_message}"
        }
        remove_field => ["message"]
      }
      mutate {
        add_field => {
          "tag" => "%{tag}"
        }
      }
    }
  }
}

Edit : If I don't remove the message field like I do in the logstash configuration, then the message field looks something like this when I view the logs in Kibana :

<30>May 15 15:13:23 devlocal e9713f013ebb[1284]: 192.168.56.110 - - [15/May/2019:15:13:23 +0200] "GET /server/status HTTP/1.0" 200 54 0.003 "-" "GuzzleHttp/6.3.3 curl/7.64.0 PHP/7.2.17" "172.30.0.2"

So the tag that I'm looking for isn't part of the message ; hence I don't know from where I can retrieve it.

Solution

Looks like the problem could be related to the log-driver that you chose. Changing the log-driver to gelf should give you access to tags, and a variety of other fields e.g. below

    {
  "_index": "logstash-2017.04.27",
  "_type": "docker",
  "_id": "AVuuiZbeYg9q2vv-JShe",
  "_score": null,
  "_source": {
    "source_host": "172.18.0.1",
    "level": 6,
    "created": "2017-04-27T08:24:45.69023959Z",
    "message": "My Message Thu Apr 27 08:31:44 UTC 2017",
    "type": "docker",
    "version": "1.1",
    "command": "/bin/sh -c while true; do echo My Message `date`; sleep 1; done;",
    "image_name": "alpine",
    "@timestamp": "2017-04-27T08:31:44.338Z",
    "container_name": "squarescaleweb_plop_1",
    "host": "plop-xps",
    "@version": "1",
    "tag": "staging",
    "image_id": "sha256:4a415e3663882fbc554ee830889c68a33b3585503892cc718a4698e91ef2a526",
    "container_id": "12b7bcd3f2f54e017680090d01330f542e629a4528f558323e33f7894ec6be53"
  },
  "fields": {
    "created": [
      1493281485690
    ],
    "@timestamp": [
      1493281904338
    ]
  },
  "sort": [
    1493281904338
  ]
}

example from: https://gist.github.com/eunomie/e7a183602b8734c47058d277700fdc2d

You would also need to send your logs via UDP instead of TCP. You can change your daemon.json to read

{
  "log-driver": "syslog",
  "log-opts": {
    "gelf-address": "udp://localhost:<PORT>"
    "tag": "{{.Name}}"
  },

  "hosts": [
    "tcp://0.0.0.0:2375",
    "unix:///var/run/docker.sock"
  ]
}

I'm not sure what port you have logstash configured to receive UDP packets, but for GELF it seems like 12201 is the default for logstash.

After the messages are sent into logstash, you can create a pipeline to extract the fields of your choice. e.g. [container_name]