Logstash grok multiple match

Question

Trying to extract some fields from the msgbody field using grok , but only the first field in the grok gets extracted.

interested Fields - corId, controller, httpStatusText and uri (These fields may not be present in every log event)

Sample Data -

2020-01-03 10:44:17,025 [93] ERROR MedServFileLogger corId=cf25b00d-1e37-4eb7-ab75-82ceeec7fdab - Exception controller= Loan action= Getmethod= GET uri= http://xxxxxxxxxx/v2/media/instance/xxxx/loans/cdb79433-32fa-4df8-b73a-e87aa89f2007/files/images-178ee8d0-fa48-4b9f-a8df-abcc9cfb1ac7.zip/entries/0b3e99f8-8af8-49a5-95b1-1537c715eb43.png?tokencreator=Encompass&tokenexpires=1578076775&token=pLCvT%2F1pBPhuFXiHKDIlB5F9feocqeq7Wxx%2FyhAz7B6DCcKeOP3YjO%2FnalfjTgXdieAmyFHEiW72Soym14oBuw%3D%3D
System.UnauthorizedAccessException: MediaTokenInvalid - A valid Token must be provided for accessing Media
2020-01-03 03:58:12,822 [37] ERROR MedServFileLogger corId=5aa9b90b-9fe6-4700-aa8f-c08be2d3f0ea - Returning controller= Health action= Getmethod= GET uri= http://localhost/v2/media/healthhttpStatusCode=503 httpStatusText=ServiceUnavailable
2020-01-03 14:13:33,987 [62] INFO  MedServFileLogger corId=2aee7503-d01e-4251-a9c3-f5f22057744b - Entering MediaTokenUtility.ValidateToken
2020-01-03 14:13:33,987 [62] INFO  EDMMediaServiceFileLogger corId=2aee7503-d01e-4251-a9c3-f5f22057744b - Entering controller= Vault action= GetFilemethod= GET uri= http://xxxxxxx.com/v2/media/vault/files/42793a9e-5123-42ae-811d-fa57b488af27?tokencreator=EDel&tokenexpires=1578352413&token=KZIc14KV0gSDlwxXhekUbMEPks3eBcc8hAHAXd7gpm8%2bDlDayNf3Vqu%2fA%2broswY3O5aOLkpq5u7fErOqtQccMA%3d%3d

Logstash Filter -

filter {

if [project] == "media_server"
{
grok {
match => [ "message", "(?m)%{TIMESTAMP_ISO8601:logtime} [(?[\d.]+)] +%{LOGLEVEL:loglevel} %{GREEDYDATA:msgbody}" ]
}

     grok {
            match => {
            break_on_match => "false"
            "msgbody" => [ "corId=%{UUID:corId}", "controller=%{SPACE}%{WORD:controller}", "httpStatusText=%{WORD:httpStatusText}", "uri=%{SPACE}%{URI:uri}" ]
            }
    }

       date {
        locale => "en"
        match => ["logtime", "YYYY-MM-dd HH:mm:ss,SSS"]
        timezone => "America/Los_Angeles"
        target => "@timestamp"

    }
    mutate
    {
            remove_field => [ "msgbody" ]
    }
}
}

Using the above configuration, only the corId field is getting extracted and all other fields are dropped. I don't see any parsing errors/failures in the logstash logs.

Followed this -

https://www.elastic.co/guide/en/logstash/5.5/plugins-filters-grok.html#plugins-filters-grok-match

Appreciate any help or guidance with this.

Thanks

Answer 1

got it working , by including the following "patterns_dir" and also "break_on_match" inside the grok filter and before the "match" stanza.

            patterns_dir => "/etc/logstash/patterns"
            break_on_match => "false"

Working Filter -

filter {

   if [project] == "media_server"
        {
        grok {
            match => [ "message", "(?m)%{TIMESTAMP_ISO8601:logtime} \[(?<threadid>[\d.]+)\] +%{LOGLEVEL:loglevel} %{GREEDYDATA:msgbody}" ]
        }

         grok {
                patterns_dir => "/etc/logstash/patterns"
                break_on_match => "false"
                match => {
                "msgbody" => [ "corId=%{UUID:corId}", "controller=%{SPACE}%{WORD:controller}", "httpStatusText=%{WORD:httpStatusText}", "uri=%{SPACE}%{URI:uri}" ]
                }
        }

           date {
            locale => "en"
            match => ["logtime", "YYYY-MM-dd HH:mm:ss,SSS"]
            timezone => "America/Los_Angeles"
            target => "@timestamp"

        }
        mutate
        {
                remove_field => [ "msgbody" ]
        }
  }
}