I had a fairly basic doubt on HFiles.
When a put/insert request is initiated, the value is first written into the WAL and then into the memstore. The values in the memstore is stored in the same sorted manner as in the HFile. Once the memstore is full, it is then flushed into a new HFile.
Now, I have read that the HFile stores the data in sorted order i.e. the sequential rowkeys will be next to each other.
Is this 100% true?
For example: I first write rows with rowkeys 1 to 1000, except rowkey 500. Assume that the memstore is now full and so it will create a new HFile, call it HFile1. Now, this file is immutable.
Now, I will write rows 1001 to 2000, then I write rowkey 500. Assume that the memstore is full and it writes to a HFile, call it HFile2.
So, is this how it happens?
If yes, then rowkey 500 is not in the HFile1, so the rowkeys in the HFiles are not in sorted order. So, is the original statement in bold correct?
So, when a read happens, how does the read happen?
HFile stores the data in sorted order i.e. the sequential rowkeys will be next to each other.
Is this 100% true?
Yes, this is 100% accurate. RowKeys with in a single HFile are always sorted.
I will write rows 1001 to 2000, then I write rowkey 500. Assume that the memstore is full and it writes to a HFile, call it HFile2.
So, is this how it happens?
Yes, now 500 gets to the top of the second HFile.
If yes, then rowkey 500 is not in the HFile1, so the rowkeys in the HFiles are not in sorted order. So, is the original statement in bold correct?
Yes, row keys with in a single HFile are always sorted. HBase periodically performs compactions which will merge multiple HFiles and rewrite's them to a single HFile, this new HFile which is a result of compaction is also sorted.
So, when a read happens, how does the read happen?
At a read time, if there are more than one HFile for a store, HBase will read that row from all the HFiles (check whether this row is there and if so read) and also from memstore. So it can get the latest data.
HBase Definitive Guide has very good explanation on how HBase Read Path works.