We want to run logstash in an ECS environment where the instances have transient lives. They might be long lived, but they aren't permanent. Thus, having logstash store the sql_last_value in the last_run_metadata_path doesn't work if the instance is removed and then brought up again. That state gets lost.
Is there a way to have that value pulled from a S3 path? Or via a DB query? I'm just looking somewhere persistent that won't vanish when the instance gets deleted and brought up again.
last_run_metadata_path => "/home/ckessel/.logstash_source_a" <-- need that to be S3 or some such
The file input just reads the saved value from a file. So the location can be anything that the Ruby IO::read function can read. That does not include s3 or a database. It needs to be a filesystem, but it can be network mounted.