I was working with JDI api in JPDA. But due to its some limitation I am stuck at some point. What are the alternatives to JDI so that I can write custom debuggers for Java programs ? Preferably Java alternatives.
JDI is implemented on top of the JVMTI native API. So, if you were willing to go to the effort of implementing your own debug agent code, you could avoid the JDI layer.
However, I doubt that it would get you any further. I suspect that the limitations in JDI are actually limitations in the JVMTI native API itself. (But you can check the JVMTI specification to see if it supports the features you desire.)
Is there an alternative to JVMTI?
AFAIK, no. At least, not for recent Oracle Hotspot releases!
(The JDPA specs acknowledge that not all Java implementations are going to implement the spec, so the possibility exists that other vendors' Java implementations do debugging differently. However, the integration between a core JRE and JVMTI represents a lot of software development effort, so it is unlikely that Oracle (or anyone else) would implement a JVMTI alternative for Hotspot.)