Let's say I run the following query precisely at datetime(2021-11-02 05:05:00)
MyTable | where ingestion_time() between (datetime(2021-11-02 05:00:00)..5m) | ..
Is it guaranteed that all the rows with their ingestion_time() falling within this time window are already available as part of the table? Experimentally I found that roughly up to 10 mins after any given time the data for that latest 5 min time window still keeps flowing. So in this example lets say that even at datetime(2021-11-02 05:10:00) , new rows are still getting populated to the table with ingestion_time() value falling between datetime(2021-11-02 05:00:00) and datetime(2021-11-02 05:05:00). Is it expected ( I guess some basic ingestion latency could be the reason ) ? And if it's expected , should we be setting some delay , say 10-15 min , in our automation to run this query for time range which was 10-15 min old?
No, there is no such guarantee. If you are looking for a way to perform exactly-once processing of all data, with no data loss guarantees, you should be using database cursors.