Issue with logstash Dissect

Question

I am having some serious problems with dissecting the below text blob in my ELK stack

This is the field -

INFO [2019-06-20 10:37:42,734] 
com.something.something.something.information.core.LoggingPiracyReporter: 
Informational request: ip_address="1.1.1.1" domain_name="domain.com" 
some_random_id="HrmwldM4DQNXoQF3AnYosJ0Mtig=" 
random_id_2="Isl/eC4ERnoLVEBMXYtWeMjwqkSKA2MPSsDnGHe4EzE=" number=1000 
timestamp=1561027064 valid_token_present=true everything_ok=true 
[Http/1.1] [8.8.8.8, 8.8.8.8, 8.8.8.8]

I have the below currently -

dissect { mapping => { "message" => '%{} ip_address="%{ip}" domain_name="% 
{name}" some_random_id="%{some_random_id}" random_id_2="%{random_id_2}" 
number="%{number}"%{}' } }

It seems to be breaking on the number field, if i remove the number it all works fine.(albeit throws a warning, but works fine and shows the fields in my kibana)

Can anyone suggest a way of getting the IP address/Domain some_random_id/random_id_2 aswell as the [http/1.1] block.

Answer 1

The quotes around %{number} in your mapping aren't present in the log you provided, which is what breaks your filter.

With this configuration:

dissect { 
    mapping => { 
        "message" => '%{} ip_address="%{ip}" domain_name="%{name}" some_random_id="%{some_random_id}" random_id_2="%{random_id_2}" number=%{number} timestamp=%{timestamp} valid_token_present=%{valid} everything_ok=%{ok} [%{http}]'
    } 
}

I'm getting this result:

{
  "ok": "true",
  "random_id_2": "Isl/eC4ERnoLVEBMXYtWeMjwqkSKA2MPSsDnGHe4EzE=",
  "message": "INFO [2019-06-20 10:37:42,734] com.something.something.something.information.core.LoggingPiracyReporter: Informational request: ip_address=\"1.1.1.1\" domain_name=\"domain.com\" some_random_id=\"HrmwldM4DQNXoQF3AnYosJ0Mtig=\" random_id_2=\"Isl/eC4ERnoLVEBMXYtWeMjwqkSKA2MPSsDnGHe4EzE=\" number=1000 timestamp=1561027064 valid_token_present=true everything_ok=true [Http/1.1] [8.8.8.8, 8.8.8.8, 8.8.8.8]\r",
  "ip": "1.1.1.1",
  "http": "Http/1.1",
  "name": "domain.com",
  "valid": "true",
  "some_random_id": "HrmwldM4DQNXoQF3AnYosJ0Mtig=",
  "timestamp": "1561027064",
  "number": "1000"
}