i know i can write c++ back end and use xul(runner) as the front end but can i use java as back end and still use xul as front end ( executing java code ) ?
You can always make an java backend which uses eg. RPC calls between the java backend and the XULrunner front end. Zimbra desktop is such and example.
Here is a small overview : http://www.zimbrablog.com/blog/archives/2009/04/zimbra-desktop-no-light-no-air.html
More info here : http://www.zimbra.com/products/desktop.html
It's free to download and the source is availible : http://wiki.zimbra.com/index.php?title=Building_Zimbra_using_Perforce#Building_Zimbra_Desktop