Running VCL in a separate thread

I now have a rather rare situation. I have an application which directly interacts with Windows' message queue. This application also runs external Lua scripts with LuaJIT. I wanted to have a debugging facility for these scripts, so I've created a plain VCL application, and then converted it into a DLL library. When the first application starts a debugging session with the library, this DLL creates a separated thread, where the whole VCL facility is initialized and run.

procedure TDebuggerThread.Execute;
begin
  Application.Initialize;
  Application.MainFormOnTaskbar := True;
  Application.CreateForm (TMainForm, MainForm);
  Application.Run;
end;

Does VCL fully supports being executed this way? To which thread will TThread.Synchronize (Proc: TThreadProc) send its message?

Inb4 "messages to VCL and to the main application will mess" - they won't because every thread has its own message queue.

Also, you may see the sources here. (Maybe) problematic library is named LuaDebugger. In place of a proper client (Core, Engine, Client) I'm currently using LuaDefaultHost, which is a rather simple console application, calling for the debugger and behaving mostly like lua.exe. With the console client, debugger works surprisingly smooth - the only problem I've encountered is that if I close the console window while the library is still used, VCL throws "Window handler is no longer valid" (in Russian :/ ). If I let the client to finish interacting with debugger the way it's supposed to, everything goes nice. Probably calling Windows.TerminateThread during unit finalization should fix that.

Solution

Your only hope is to create the thread, and then load the DLL from that thread. So, to be as clear as possible, you create the thread and then from code executing within that thread, you call LoadLibrary to load the DLL.

The VCL has to be run out of the thread that loads the DLL. The VCL initialization happens during the initialization of the DLL and that determines which thread is the VCL main thread. The VCL main thread is the thread which initializes the VCL, the thread which loads the DLL.

You'll likely have to keep a clear head with this entire approach because you'll have two GUI threads, two message pumps, in a single process. Showing a modal window involves disabling the windows on both GUI threads.

I cannot be sure that this general approach (two GUI threads in the same process, one of which is a VCL thread) will work, never having done it. However I think there's a good chance it will fly.

You also ask a quite specific question:

To which thread will TThread.Synchronize (Proc: TThreadProc) send its message?

The answer is always the thread which initialized the module. So for an executable this is the main thread of the process. For a DLL the thread which initialized the module is the thread which called LoadLibrary, the thread which executes the initial call to DllMain, the thread which executes the DLL units' initialization code. This is known in the RTL/VCL as the module's main thread. It is the thread whose ID is given by System.MainThreadID.

To prove the point, in case you don't take my word for this, here's a little demonstration.

Executable

program DllThreading;

{$APPTYPE CONSOLE}

uses
  Classes, Windows;

type
  TMyThread = class(TThread)
  protected
    procedure Execute; override;
  end;

procedure TMyThread.Execute;
var
  lib: HMODULE;
  proc: procedure; stdcall;
begin
  lib := LoadLibrary('dll.dll');
  proc := GetProcAddress(lib, 'foo');
  proc();
  Sleep(INFINITE);
end;

begin
  Writeln('This is the process main thread: ', GetCurrentThreadId);
  TMyThread.Create;
  Readln;
end.

DLL

library Dll;

uses
  Classes, Windows;

type
  TMyThread = class(TThread)
  private
    procedure DoStuff;
  protected
    procedure Execute; override;
  end;

procedure TMyThread.DoStuff;
begin
  Writeln('This is the thread which executes synchronized methods in the DLL: ', GetCurrentThreadId);
end;

procedure TMyThread.Execute;
begin
  Writeln('This is the thread created in the DLL: ', GetCurrentThreadId);
  Synchronize(DoStuff);
end;

procedure foo; stdcall;
begin
  TMyThread.Create;
  CheckSynchronize(1000);
end;

exports
  foo;

begin
  Writeln('This is the initialization thread of the DLL: ', GetCurrentThreadId);
end.

Output

This is the process main thread: 2788
This is the initialization thread of the DLL: 5752
This is the thread created in the DLL: 6232
This is the thread which executes synchronized methods in the DLL: 5752