Why elastic index storage size increase while updated index not insert?

Question

I write to index from oracle via logstash.

Output is :

   elasticsearch {
                index => "****"
                document_type => "****"
                document_id => "%{my_computed_id}"
                hosts => "localhost:9200"
 }

But when I check the index management , the Docs Count is same but storage size increase.

I want to update my docs so why the storage size increase?

thanks in advance

Answer 1

This has happened because of how elasticsearch updates the documents.

When you update a document elasticsearch doesn't simply override the document. It marks the old document as deleted and creates a new document with the the fields present in current document.

At the time of searching, documents that are marked as deleted are not included in the search.

Also you need to understand how data is stored at the backend in lucene. On the backend one shard of elasticsearch is one index of lucene. Each lucene index has multiple segments in it.

When segment size becomes large multiple segments merge to form a new big segment.

When segments merge to form a new segment, documents marked as deleted are not included in new segment.

So your data size will increase temporarily, but eventually it will come down.

Here is a link : https://www.elastic.co/blog/lucenes-handling-of-deleted-documents