Search code examples
logstashlogstash-grok

grok expression to pull string with in brackets

Am currently trying to parse log messages using grok. below is the sample message

[de4131185bb28bcd5f3019b168106682] [172.69.62.54] [/v2/user.json] Returned lineup_key for user OAuth [ OAuth realm="", oauth_consumer_key="qQRye5YLMtdw2TNmoiayMptwdbW29y3pTlaIj0lx", oauth_token="ArFbNvrxntvYIbA2QArJnCmbGCBQe3P5I1YHX8TX", oauth_signature_method="HMAC-SHA1", oauth_signature="o1DhpgEX7QyCDjDvPceHzJrm7h4%3D", oauth_timestamp="1529081887", oauth_nonce="0C6F8BAC-B3E3-44F3-A3BD-F04CF7815580", oauth_version="1.0" ] : [ MjE2LDI1OA== ]

The grok expression being used is below

UMSAPPLOGDATA ^\[%{WORD:id}\]\s\[%{IP:ip}\]\s\[%{URIPATHPARAM:uri}\]%{GREEDYDATA:logMessage}

Now i want to extract the entire Oauth piece below into separate variable called oauth_parameters. 

[ OAuth realm="", oauth_consumer_key="qQRye5YLMtdw2TNmoiayMptwdbW29y3pTlaIj0lx", oauth_token="ArFbNvrxntvYIbA2QArJnCmbGCBQe3P5I1YHX8TX", oauth_signature_method="HMAC-SHA1", oauth_signature="o1DhpgEX7QyCDjDvPceHzJrm7h4%3D", oauth_timestamp="1529081887", oauth_nonce="0C6F8BAC-B3E3-44F3-A3BD-F04CF7815580", oauth_version="1.0" ]

How can i extend my existing grok expression for the same.

appreciate you help regarding the same.

Solution

  • You can write a custom grok pattern in the following format,

    (?<field_name>the pattern here)

    Your pattern would then be,

    \buser OAuth (?<outh_parameters>(.*)oauth_version="\d{1,2}.\d{1,2}" ])

    It will output,

    {
  "outh_parameters": [
    [
      "[ OAuth realm="", oauth_consumer_key="qQRye5YLMtdw2TNmoiayMptwdbW29y3pTlaIj0lx", oauth_token="ArFbNvrxntvYIbA2QArJnCmbGCBQe3P5I1YHX8TX", oauth_signature_method="HMAC-SHA1", oauth_signature="o1DhpgEX7QyCDjDvPceHzJrm7h4%3D", oauth_timestamp="1529081887", oauth_nonce="0C6F8BAC-B3E3-44F3-A3BD-F04CF7815580", oauth_version="1.0" ]"
    ]
  ]
}