Search code examples
javaelasticsearchflume

Flume + ElasticSearch Sink TTL

I've got a question about the TTL in elasticsearch sink of apache flume

I've working on elastic search + flume integration. I'm using elasticsearch version 1.4.1 and flume version 1.5.2 Both are running locally on my machine

In Flume My ElasticSearch Sink is configured as follows:

agent.sinks.elasticSearchSink.type = org.apache.flume.sink.elasticsearch.ElasticSearchSink
agent.sinks.elasticSearchSink.channel = fileChannel
agent.sinks.elasticSearchSink.hostNames=localhost:9300
agent.sinks.elasticSearchSink.indexName=platform
agent.sinks.elasticSearchSink.indexType=platformtype
agent.sinks.elasticSearchSink.ttl=1m
agent.sinks.elasticSearchSink.batchSize=1000
agent.sinks.elasticSearchSink.serializer=org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer

Note, there is a ttl of 1m (1 minute) for the sake of test.

I see that events get added into ES but are not deleted after a minute. The "mapping" query also doesn't show that there is a TTL. I know that the TTL is disabled by default, so I can enable it like this:

>> PUT: http://localhost:9200/_all/platformtype/_mapping
with body:
{"platformtype" : {"_ttl" : {"enabled" : true, "default" : "2m"}}}

Note, that now its 2 minutes TTL (just to be different with the sink definitions)

So, now if I add other events they get deleted after 1 minute...

So could someone shed some light how exactly this should work? Is it a bug or I have to manually enable the TTL?

Thanks

Solution

  • Well, It turned out that it works like this: The TTL has to be enabled via the mapping API in elasticsearch. If its not done, the TTL sent from Flume just gets ignored.

    Now, The TTL enabled at the elasticsearch level with the presented definitions work as follows:

    "Index 'platformtype' will have the TTL with the default value of 2 minutes. So, if I disable the TTL at the flume level, the messages will be deleted in 2 minutes (the TTL is not specified in the event sent from flume, so the default value comes into play).

    Alternatively, if there is an explicit TTL value supplied by flume, it will take precedence over a default TTL definition, so in this case the corresponding record will be deleted indeed in 1 minute as I've mentioned.

    Hope this will help to somebody. Thanks, the question is closed :)