What metrics can I use to validate and test RankNet in the RankLib library in the Lemur Project?

Question

I am currently using the RankLib implementation of the RankNet algorithm (-ranker 4) with a held-out set. I am using the jar file in terminal to run this.

The documentation stipulates:

metric2t (e.g. NDCG, ERR, etc) only applies to list-wise algorithms (AdaRank, Coordinate Ascent and LambdaMART). Point-wise and pair-wise techniques (MART, RankNet, RankBoost), due to their nature, always use their internal RMSE / pair-wise loss as the optimisation criteria.

However, when I set the 'metrics2t' to ERR@10 or NDCG@10, it starts to train and validate on my chosen metric rather that 'RMSE'.

This is part of the table outputted when I run RankNet with ERR@10.

Is there something that I am missing as this seems to be a contradiction to me.

Thanks.

Answer 1

I am not sure, but, I think even if it prints the result for those metrics, it is not optimizing for them.

The library's developers simply left it there, as for other methods it is common to use one of those metrics for validation. And there is no option to simply turn of the computing off the metrics during training.

Right now I am training a RankNet model, and it seems that ERR@10 for training and validation data is actually increasing, while the "% mis-ordered pairs" is decreasing.