I have set up a new AWS Kinesis Firehose stream and I'd like to create a new index on a weekly basis.
For that, I should use Index Rotation setting when configuring the stream.
But, do I have to the new index every weekend for the next upcoming week?
If not (hopefully not), how does Firehose knows what mapping to use? does it use the mapping defined in the index that I specified in the Index setting?
More over, lets say I have old data, can I make Firehose to create an index with the relevant timestamp according to the dates in specified in my old data?
Thank you !
Well, apparently the answer is yes, but in a bad way.
If Firehose pushes data to a new index, that is not pre-defined with a mapping, that data is ingested to Elasticsearch and a mapping is created automatically for you.
This is really bad.
You should auto create an index 1-2 hours prior the event.
I'll post a Lambda function and its configuration for doing that automatically.