Search code examples
statisticstokenizeranking

How to compute word rank in perl

I have a program where I read in a document and then put all the words in a hash delete repeats and add to frequency of that word.

So for example:

KEY: VALUE: dog 2 cat 4 rat 1

Now I was told I need to compute the rank of each word and print those stats. What does this mean exactly? What type of math do I need to be looking at? If someone could point me to a document talking about word rank that could help.

Thanks

Solution

  • Ranking is simply ordering so that the most frequent word has rank 1. Take a look at Zipf's law for how we expect words to behave with respect to their frequency rank in a suitably large corpus.