Identifying when logstash jdbc SQL statement has completed execution

Question

I've set up a logstash pipeline watching a directory for logstash.confs with a jdbc input and elasticsearch output.

I enabled .logstash_jdbc_test_last_run, which logs the time when query is executed:

--- 2019-08-23 15:26:42.847349000 Z

When running logstash on the command line, my pipeline successfully processes a logstash conf file:

[2019-08-23T15:26:48,168][INFO ][logstash.inputs.jdbc     ] (5.250782s) select S0.* from mytable S0

I'm trying to have my application notified that logstash has finished processing a logstash conf and the data is available in ES. Is there a way to write the timestamp when the query has completed to a file (for use as a flag in my application)?

Answer 1

Ultimately I was able to get the desired result by leveraging multiple pipelines Thanks @apt-get_install_skill, you led me down the right path!

pipelines.yml:

- pipeline.id: logstash
  pipeline.workers: 1
  path.config: "/tmp/logstash*.conf"
  queue.type: persisted
- pipeline.id: postprocessing
  pipeline.workers: 1
  path.config: "/tmp/postprocessing*.conf"

logstash.conf:

input {
   jdbc {
     ...
   }
}
output   {
    elasticsearch {
      ...
    }
    pipeline { send_to => [postProcessing] }
}

postprocessing.conf, which uses the output from the logstash pipeline:

input {
    pipeline { address => postProcessing }
}
output {
    file{
        path => "/tmp/finishedflag"
        codec => "dots"
    }
}

Codec dots as I don't care about the data itself.

When it runs, it will first run the logstash pipeline, and when it completes it runs the postprocessing pipeline.