A while ago we added Python scripting to a Wpf application using IronPython. At first it was only 'slave' in the sense that a script was invoked by a button click for instance and then just ran to completion returning control to Wpf. Later on we added 'master' scripting: the script runs in it's own thread, and controls the rest of the application. That was quite challenging but after a while and with help of existing SO content we got it working, seemingly. Never really used it though, until now, and unfortunately it turns out it does not work properly. Core cause is that although there are two seperate STA threads (the main Wpf one and one for the script), and hence two different Dispatcher instances, the main thread seems to get blocked because the script thread is in a loop waiting for the main thread to complete (in response to a button click processed on the script thread and starting events on the main thread). The whole point of using two threads with seperate ui windows was of course this wouldn't happen. What is going on?
update It is reproducable with minimal code, so I'm linking to that instead of posting pseudo-code here. While creating the code I found that when the window created by the script thread is not embedded (set MainWindow.hostedWin = false) the deadlock does not occur and everything behaves as expected.
in response to comments So there are 3 threads of concern coming into play. Let's call them Python, Ui and Process. Python starts Process and waits for it to complete. Process calls Invoke on Ui. Which shouldn't be doing anything at that point: after all, it's Python that is blocking, not Ui, and the whole point of this construction is that Ui shouldn't have to interact with Python. Well, except that it does somehow. Which is the culprit. In the deadlock, Ui sits at PresentationFramework.dll!System.Windows.Interop.HwndHost.OnWindowPositionChanged(System.Windows.Rect rcBoundingBox) + 0x82 bytes and Process sits at WindowsBase.dll!System.Windows.Threading.DispatcherOperation.DispatcherOperationEvent.WaitOne() + 0x2f bytes and Python is just at Thread.Sleep.
What is going on here, and how to fix it?
I'll keep it short, very few odds that this answer is going to make you happy. It is a three-way deadlock. The most severe one in the interaction between the main thread and PythonThread. This deadlock occurs in the Windows kernel, the NtUserSetWindowPos() call cannot progress. It is blocked, waiting for the WM_LBUTTONUP callback notification on the PythonThread to finish running.
This deadlock is caused by your WpfHwndEmbedHost hack. Turning a top-level window owned by another thread or process into a child window is an appcompat feature that was meant to support Windows 3.x programs. A Windows version that did not yet support threads and where having one task embedding another task's window wasn't a problem. A WPF window isn't exactly much like such a window, to put it mildly. Otherwise a well-known troublemaker, for one the reason that embedding Acrobat Reader in a browser window works so very poorly. Not turning on the WS_CHILD style flag ought to bring relief, but WPF isn't happy about that. Simply setting hostedWin to false solves the problem.
The other deadlock is the one I warned you about, the interaction between the main thread and the ProcessThread. Dispatcher.Invoke() is dangerous, it deadlocks because the main thread is stuck in the kernel. Using Dispatcher.BeginInvoke() solves the problem. Partly, you still have the main thread go catatonic for 5 seconds.
The most severe problem is the kernel lock, that's going to bite in many other ways. You are going to have to keep it a separate window to avoid it. Not good news, I'm sure.