On the GWT overview page it reads:
The GWT developer plugin spans the gap between Java bytecode in the debugger and the browser's JavaScript.
Thanks to the GWT developer plugin, there's no compiling of code to JavaScript to view it in the browser. You can use the same edit-refresh-view cycle you're used to with JavaScript, while at the same time inspect variables, set breakpoints, and utilize all the other debugger tools available to you with Java. And because GWT's development mode is now in the browser itself, you can use tools like Firebug and Inspector as you code in Java.
And the another SO Question and Answer How GWT code runs in development code mentions the JS code is extracted and put into the browser to be evaluated and result is sent back to the java.
What's the exact protocol of this process? Any documentation (not the very high level ones)? Where are the locations in source code? How do you track this interface in the browser or in the java vm (firebug, or java debugger)?
Edit: Artem below answered how it looks like from the lowest layer on the Java side. What's the higher layers, if you know? What's on the browser side?
Everything's documented in the wiki actually: https://code.google.com/p/google-web-toolkit/wiki/DesignOOPHM
If you want to write a client replacement, you can use the C++ code from https://gwt.googlesource.com/gwt-plugins/+/master/common or in Java the BrowserChannelClient class. Note that there's also an unfinished wireshark packet dissector in https://gwt.googlesource.com/gwt-plugins/+/master/wireshark