Grokking context of PHP monolog message into ELK

Question

I want to be able to search in Kibana on the fields that I log as context in Monolog.

I mean that if I $log->info("Request", $visitDetails); where $visitDetails is an array then I'd like for Logstash to grok this for Elastic so that the array key/values are indexed and searchable.

My PHP looks like this:

// "logstash" is a host defined by docker-compose
$handler = new SocketHandler('logstash:9001', Logger::DEBUG);
$log->pushHandler($handler);
$log->pushHandler(new StreamHandler(__DIR__ . '/test.log', Logger::DEBUG));


// log the details of the user visit
$visitDetails = [
    'ip' => $_SERVER['REMOTE_ADDR'],
    'method' => $_SERVER['REQUEST_METHOD'],
    'uri' => $_SERVER['REQUEST_URI'],
    'agent' => $_SERVER['HTTP_USER_AGENT'],
    'referer' => $_SERVER['HTTP_REFERER'] ?? 'not set'
];
$log->info("Request", $visitDetails);

and my Logstash config looks like this:

input {
    tcp {
        port => 9001
    }
}

filter {
    grok {
        match => { "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:env}\.%{DATA:severity}: %{GREEDYDATA:message} \{%{GREEDYDATA:context}\} \[\]" }
    }
}

output {
    stdout {}
}

I'm using https://grokdebug.herokuapp.com/ to help work out the grok pattern and can now isolate "context" as a JSON string.

How do I create a filter that will expand that into key/value pairs for Elastic?

I think part of the problem is that the braces {} around the json string are being taken away. I'm struggling to make a non-capturing group for them in grok.

Answer 1

I rather used the regex feature of Grok (powered by https://github.com/kkos/oniguruma/blob/master/doc/RE).

This let me use regex, with this syntax (?<field_name>the pattern here) and then I used a non-capturing group for the opening brackets.

This seems to do what I want it to:

filter {
    grok {
        match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:env}\.%{DATA:severity}: %{GREEDYDATA:message} (?<context>(\{.*?\})) \[\]" }
    }
}

filter {
      json {
        source => "context"
      }
    }