Supposed given a query, our system return 10 recommendation result
doc1, doc2, ... doc10
if the ground truth says relevant document says the relevant document is doc100 (which is in position 100), does the reciprocal rank should be 1/100 ? I think it's hard to push the system to return 100 document.
But if i only include recommendation result which had the relevant document in position below top10, I think that is a Naive approach. And i don't really understand the concept of MRR@10, MRR@5, etc.
Can someone please explain to me? Thank you
The MRR in that case is defined to be zero.
The reasoning is this: imagine you didn’t have any more documents; how would you tell the difference between it being on the next page or not being returned at all?
The goal of these low rank MRR is to model the first page or a website that can only fit three recommendations. To the user there’s no difference between documents that appear just after the cutoff and a hundred ranks deeper.