Grok filter for selecting and formatting certain logs lines

I am writing up a grok filter for parsing my application log which is unstructured. What i need is to look for certain lines and generate output in a specific format. e.g below are my logs

2018-05-07 01:19:40 M :Memory (xivr = 513.2 Mb, system = 3502.0 Mb, physical = 5386.7 Mb), CpuLoad (sys = 0%, xivr = 0%)
2018-05-07 01:29:40 M :Memory (xivr = 513.2 Mb, system = 3495.3 Mb, physical = 5370.1 Mb), CpuLoad (sys = 0%, xivr = 0%)
2018-05-07 05:51:19 1 :Hangup call
***2018-05-07 05:51:22 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx***]
2018-05-07 05:51:30 24 :Hangup call
***2018-05-07 05:51:34 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx]***
2018-05-07 00:31:21 45 :Device Dialogic Digital dxxxB12C1 [gc60.dev - Dialogic (SDK 6.0) ver 3.0.702:11646] (ThreadID: 1FF0, DriverChannel: 44)
2018-05-07 00:31:22 40 :Device Dialogic Digital dxxxB10C4 [gc60.dev - Dialogic (SDK 6.0) ver 3.0.702:11646] (ThreadID: 1B2C, DriverChannel: 39)

I need to enter only lines highlighted with *** in below format in my Kibana: Other lines should be simply ignored

Logtimestamp: 2018-05-07 05:51:22
Channel_id: 24
Source_number: 71840746
Destination_Number: 91783028

How can this be achieved?

Solution

You can explicitly write whatever is unique about that particular pattern, and use pre-defined grok patterns for the rest.

In your case, the grok pattern would be,

%{TIMESTAMP_ISO8601:Logtimestamp} %{NUMBER:Channel_id} :Answer call from %{NUMBER:Source_number} for %{NUMBER:Destination_Number} %{GREEDYDATA:etc}

It will only match following pattern,

2018-05-07 05:51:34 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx]

Explanation

The syntax for a grok pattern is %{SYNTAX:SEMANTIC}.

In your filter,

%{TIMESTAMP_ISO8601:Logtimestamp} matches 2018-05-07 05:51:34
%{NUMBER:Channel_id} match 24
:Answer call from matches the string literally
%{NUMBER:Source_number} matches 71840746
%{NUMBER:Destination_Number} matches 91783028
%{GREEDYDATA:etc} matches rest of the data i.e. [C:\xivr\es\IVR-Dialin.dtx]

in that order.

Output:

{
  "Logtimestamp": [
    [
      "2018-05-07 05:51:22"
    ]
  ],
  "Channel_id": [
    [
      "24"
    ]
  ],
  "Source_number": [
    [
      "71840746"
    ]
  ],
  "Destination_Number": [
    [
      "91783028"
    ]
  ],
  "etc": [
    [
      "[C:\\xivr\\es\\IVR-Dialin.dtx***]"
    ]
  ]
}

You can test it here.

Hope it helps.