How to extract and visualize values from a log entry in OpenShift EFK stack

I have an OKD cluster setup with EFK stack for logging, as described here. I have never worked with one of the components before.

One deployment logs requests that contain a specific value that I'm interested in. I would like to extract just this value and visualize it with an area map in Kibana that shows the amount of requests and where they come from.

The content of the message field basically looks like this:

[fooServiceClient#doStuff] {"somekey":"somevalue", "multivalue-key": {"plz":"12345", "foo": "bar"}, "someotherkey":"someothervalue"}

This plz is a German zip code, which I would like to visualize as described.

My problem here is that I have no idea how to extract this value.

A nice first success would be if I could find it with a regexp, but Kibana doesn't seem to work the way I think it does. Following its docs, I expect this /\"plz\":\"[0-9]{5}\"/ to deliver me the result, but I get 0 hits (time interval is set correctly). Even if this regexp matches, I would only find the log entry where this is contained and not just the specifc value. How do I go on here?

I guess I also need an external geocoding service, but at which point would I include it? Or does Kibana itself know how to map zip codes to geometries?

A beginner-friendly step-by-step guide would be perfect, but I could settle for some hints that guide me there.

Solution

It would be possible to parse the message field as the document gets indexed into ES, using an ingest pipeline with grok processor.

First, create the ingest pipeline like this:

PUT _ingest/pipeline/parse-plz
{
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{POSINT:plz}"
        ]
      }
    }
  ]
}

Then, when you index your data, you simply reference that pipeline:

PUT plz/_doc/1?pipeline=parse-plz
{
  "message": """[fooServiceClient#doStuff] {"somekey":"somevalue", "multivalue-key": {"plz":"12345", "foo": "bar"}, "someotherkey":"someothervalue"}"""
}

And you will end up with a document like the one below, which now has a field called plz with the 12345 value in it:

{
  "message": """[fooServiceClient#doStuff] {"somekey":"somevalue", "multivalue-key": {"plz":"12345", "foo": "bar"}, "someotherkey":"someothervalue"}""",
  "plz": "12345"
}

When indexing your document from Fluentd, you can specify a pipeline to be used in the configuration. If you can't or don't want to modify your Fluentd configuration, you can also define a default pipeline for your index that will kick in every time a new document is indexed. Simply run this on your index and you won't need to specify ?pipeline=parse-plz when indexing documents:

PUT index/_settings
{
  "index.default_pipeline": "parse-plz"
}

If you have several indexes, a better approach might be to define an index template instead, so that whenever a new index called project.foo-something is created, the settings are going to be applied:

PUT _template/project-indexes
{
  "index_patterns": ["project.foo*"],
  "settings": {
     "index.default_pipeline": "parse-plz"
  }
}

Now, in order to map that PLZ on a map, you'll first need to find a data set that provides you with geolocations for each PLZ.

You can then add a second processor in your pipeline in order to do the PLZ/ZIP to lat,lon mapping:

PUT _ingest/pipeline/parse-plz
{
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{POSINT:plz}"
        ]
      }
    },
    {
      "script": {
        "lang": "painless",
        "source": "ctx.location = params[ctx.plz];",
        "params": {
          "12345": {"lat": 42.36, "lon": 7.33}
        }
      }
    }
  ]
}

Ultimately, your document will look like this and you'll be able to leverage the location field in a Kibana visualization:

{
  "message": """[fooServiceClient#doStuff] {"somekey":"somevalue", "multivalue-key": {"plz":"12345", "foo": "bar"}, "someotherkey":"someothervalue"}""",
  "plz": "12345",
  "location": {
    "lat": 42.36, 
    "lon": 7.33
  }
}

So to sum it all up, it all boils down to only two things:

Create an ingest pipeline to parse documents as they get indexed
Create an index template for all project* indexes whose settings include the pipeline created in step 1