If I am at site abc.com and I go to my site def.com which hosts a silverlight app. How can I find out which domain I just came from within my silverlight app?
You can't. You will have to use Javascript for that and even then you won't be able to see from which site the user came, you can just go back to the previous page.
You can do it on the server side by checking the web server logs.