I just updated from ELK 6.7.0 to 7.4.0 (via 6.8.0). After I started my 7.4.0 instance, logstash started putting the data into an index called "logstash" instead of "logstash-2019.10.24" like it used to.
I updated my output plugin to look like
output {
elasticsearch {
hosts => ["localhost"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
I didn't used to have the "index" part, but I tried that to see if it would help, restarted logstash, and it's still putting data into the "logstash" index.
How can I get logstash to use the date-stamped indexes again?
(I am running just a single-node in docker: https://github.com/kenwdelong/elk-docker/tree/ELK-7.4.0)
Strangely, if I spin up the same tagged docker image on a clean machine, and shove some data into it using the test script on my github page, it works fine and I get the date in the index name.
I finally figured this out. I had an installed template called "logstash" with the index pattern "logstash-*". This template was from a long time ago and still had the "defaults" key directly underneath "mappings". Apparently this prevented ES 7 from creating the index, so Logstash somehow fell back to the "logstash" index. I had to drop the logstash template, then recreate it without the "defaults" key. Also, I ran into some problems where records were not being indexed because of ECS - ES was assuming that certain fields (like userAgent) were one type when actually everything I have was set up to be different type. So I had to iron out those problems too.