Search code examples
algorithmsortingranking

Ranking from pairwise comparisons


Imagine I have a very long list of images and I want to rank them in order of how 'good' people think they are.

I don't want to get users to assign scores to images outright (1 - 10 etc) and order by that, I'd like to try something new.

What I was thinking would be an interesting way of doing it is:

  • Show a user two random images, they pick the better one
  • Collect lots of 'comparisons'
  • Use all the comparisons to come up with some ordering

Turns out this is used regularly, for example (using features, not images), this appears to be the way Uservoice's Smartvote works.

My question is whether there's a good known way to take this long list of comparisons and build a relative ranking for all the images from them but not to the level of complexity found in the research papers.

I've read a bunch of lectures and research papers but I was wondering if there was any sample code out there people might recommend?


Solution

  • Seems like you could just get some kind of numerical ranking system and then just sort based on that. Just borrow the algorithm from a win/loss sport, or chess, and treat each image comparison as a bout.

    Did some looking, here's some sample code of what an algorithm like that looks like in Java

    And here's a library you can borrow in python

    If you search ELO you'll find a version of it in just about any language. Once you get your numerical image rankings, you can sort them any way you like. There are probably other ranking algorithms you could look into for win/loss competition, that was just the first that came up when I googled chess ranking.