elastic logstash multiline + key value when one of the key(Message) is more then one line

Question

I have the following log file,every Logstash record is supposed to contain multiline lines that end with dash signs '----' ,from the example log you can see that I have four keys, the last key is 'Message' . I have configured the following config file that handles all the lines except the line that starts with 'Message :' As you can see from the log in some of the cases, the 'Message :' contains more than one line, and for these cases, I need that all the lines coming after the 'Message :' will be part of the 'Message :' value and not a separate lines .

Please help me fixing this issue.

log file (input.log)

Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Return code : EXCI_NO_ERROR 0
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Abend code : 1564
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Buffer received from xxx
  line1
  line2
  line3
  line4
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Return code : EXCI_NO_ERROR 0
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Abend code : 1564
------------------------------------------

config file

input {
  file {
         path => "/etc/logstash/input.log"
                 start_position => "beginning"
                 sincedb_path => "/dev/null"

         codec => multiline {
         # pattern = This says that any line not starting with a '-----' should be merged with the previous line
           pattern => "^-----"
           negate => true
           what => "previous"
        }
  }
}

filter {

      kv {
        #source => "message"
        field_split => "\n"
        value_split => ":"
        }

      }


output {

        file {
         path => "/etc/logstash/output.log"
        }

        #stdout { codec => rubydebug }
}

Answer 1

Tldr;

Take a step back and use other filters type.

• dissect
• mutate

You will find below one possible solution.

Solution

1. Mutate, remove the ----\n from the message
2. Drop empty message
3. Dissect the message into the necessary fields

input {
  file {
    path => "/tmp/input.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
    codec => multiline {
      # pattern = This says that any line not starting with a '-----' should be merged with the previous line
        pattern => "^-+$"
        negate => true
        what => "previous"
    }
  }
}

filter {
  mutate { gsub => ["message", "^-+(\n)?", ""] }
  if [message] == ""{
    drop {}
  }
  dissect {
    mapping => {
      "message" => "Timestamp :%{Timestamp}
User :%{User}
Type :%{Type}
Message :%{Message}"
    }
  }
}

output {
  stdout { codec => rubydebug }
}