Logstash can't add fields?

Question

I have been using logstash to read some DB restore logs. Here is some lines of sample records.

07/08/2016  6:33:22.50: START restore database                      
SQL2540W  Restore is successful, however a warning "2539" was encountered 
during Database Restore while processing in No Interrupt mode.
07/08/2016  6:33:28.93: END restore database                        
SQL4406W  The DB2 Administration Server was started successfully.
07/08/2016  6:35:35.29: END restart server                          
connect reset
DB20000I  The SQL command completed successfully.
07/08/2016  6:35:38.48: END p:\s6\source\system\CMD\res_uw.cmd      

Here is the filter part of my conf file.

if ([message] =~ /Backup successful/){
    grok{
        match => {"message" => ['%{GREEDYDATA:Message}'] }
    }
    mutate {
        add_tag => "send_to_es"
        add_field => {"Timestamp" => "%{GREEDYDATA:DATETIME}"}
    }
}
if ([message] =~ /warning "2539"/){
    grok{
        match => {"message" => ['%{GREEDYDATA:Message}'] }
    }
    mutate {
        add_tag => "send_to_es"
        add_field => {"Timestamp" => "%{GREEDYDATA:DATETIME}"}
    }
}
if ([message] =~ /(END p:|END P:)/){
    grok{
        match => {"message" => ['%{GREEDYDATA:DATETIME}:%{SPACE}END%{SPACE}%{GREEDYDATA:Mis}'] }
        remove_field => "%{GREEDYDATA:Mis}"
    }
    mutate {
        add_tag => "send_to_es"
    }
}   

I want to add the data "DATETIME" extracted from the last line of my record to message to other message to index at the same time. However, it could not add the field successfully. The output will become

      "message": "SQL2540W  Restore is successful, however a warning \"2539\" was encountered \r\r",
      "@version": "1",
      "@timestamp": "2016-07-12T02:28:52.337Z",
      "path": "C:/CIGNA/hkiapp67_db_restore/res_uw.log",
      "host": "SIMSPad",
      "type": "txt",
      "Message": "SQL2540W  Restore is successful, however a warning \"2539\" was encountered \r\r",
      "Timestamp": "%{GREEDYDATA:DATETIME}",
      "tags": [
        "send_to_es"
      ]

How could I solve this?

Answer 1

Logstash, when receiving a line, does not have knowledge of any other line. You'll have to use a multiline codec/filter to regroup all the lines you need with a line with the date. Then you use the grok filter to extract the date and add it to your document.

The configuration of the multiline codec/filter will look like this:

multiline {
  pattern => "%{DATE}"
  negate => "true"
  what => "next"
}

With this, all the lines not beginning with the pattern DATE will be joined with the next line.