I'm trying to find out fast motion estimation algorithm used in vp9, kindly help me out. As no documentation is available although it's open source but I couldn't find anything relevant.
As with typical video standards, there is no motion estimation algorithm in VP9, the standardized parts are the bitstream and how to decode it. Of course the encoders implement some motion estimation algorithm(s) (usually configurable so the user can choose their speed/quality trade-off), but since the standard doesn't cover encoders that is not part of VP9. For the decoder it does not matter how motion vectors were chosen, it only matters what the result was.
You can get the latest version of the standard from this website.
In libvpx in vp9_mcomp.c, it is visible which algorithms that specific encoder uses, which includes several diamond searches (with varying accuracy/time trade-offs including N-step diamond search), two hexagon-based searches, square search, and even exhaustive search. There's integral projection motion estimation in it too but it seems to be used only in a special case.