filter to extract data from multiple lines using grok

Question

i am new to logstash i am try to find pattern to extract data from this log messages where i enable the pattern in filebeat.yml to read from date to next occurence of date.

2018-05-21 14:49:12
Mode:Managed  Frequency:2.457 GHz  Access Point: 88:D7:F6:68:C1:78   
Bit Rate=144.4 Mb/s   Tx-Power=22 dBm   
Retry short limit:7   RTS thr:off   Fragment thr:off
Power Management:on
link Quality=65/70  Signal level=-45 dBm  
Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
Tx excessive retries:0  Invalid misc:217   Missed beacon:0





    grok{
 timeout_millis => 60000
match=>["message", "%{TIMESTAMP_ISO8601:mytimestamp}%{SPACE:ip}%{GREEDYDATA:val}%{SPACE:ip}%{GREEDYDATA:val}%{SPACE:ip}%{GREEDYDATA:val}%{SPACE:ip}%{GREEDYDATA:val}%{SPACE:ip}%{GREEDYDATA:val}%{SPACE:ip}%{GREEDYDATA:val}(?<powerlevel>(?<=Signal level\=).*?(\s))"]
}

this gives _groktimeout

filter {
    grok {
    match => ["message", "%{TIMESTAMP_ISO8601:mytimestamp}",
       "message", "(?<powerlevel>(?<=Signal level\=).*?(\s))"]

    }

this gives only timestamp please can some one help me to fetch timestamp and signal level from this log

Answer 1

You need to match data between your date and Signal level as well. This can be done using GREEDYDATA pattern. Besides, you also need to match all the spaces and \n characters as well.

Have a look at the following,

%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}(?m)%{GREEDYDATA:irrelevant_data}Signal level=%{NUMBER:Signal level}

It will match both date and Signal level,

output,

{
  "YEAR": [
    [
      "2018"
    ]
  ],
  "MONTHNUM": [
    [
      "05"
    ]
  ],
  "MONTHDAY": [
    [
      "21"
    ]
  ],
  "TIME": [
    [
      "14:49:12"
    ]
  ],
  "HOUR": [
    [
      "14"
    ]
  ],
  "MINUTE": [
    [
      "49"
    ]
  ],
  "SECOND": [
    [
      "12"
    ]
  ],
  "irrelevant_data": [
    [
      "\nMode:Managed  Frequency:2.457 GHz  Access Point: 88:D7:F6:68:C1:78   \nBit Rate=144.4 Mb/s   Tx-Power=22 dBm   \nRetry short limit:7   RTS thr:off   Fragment thr:off\nPower Management:on\nlink Quality=65/70  "
    ]
  ],
  "Signal": [
    [
      "-45"
    ]
  ],
  "BASE10NUM": [
    [
      "-45"
    ]
  ]
}

Your grok filter will become,

filter {
    grok {
    match => ["message", "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}(?m)%{GREEDYDATA:irrelevant_data}Signal level=%{NUMBER:Signal level}"]
  }
}