Stop being topmost window

My window should be on top of a specific "target" window that I don't have control over.

When the target window is activated, I call SetWindowPos with HWND_TOPMOST to place my window on top of it while the target can still be the active window.

When the target window is no longer the foreground window, I want my window to still be on top of the target window, but no longer topmost, so other windows are not covered by it.

Two ideas I had:

Call SetWindowPos with hWndInsertAfter to be the just activated window. This fails when the just activated window is topmost, because my window then does not lose the topmost status. Another issue with this: If the just activated window is the desktop, then my window is placed below the target window.
Call SetWindowPos with HWND_NOTOPMOST to lose the topmost status. However, this brings my window to the top of all non-topmost windows, so it covers the just activated window. To fix this I have to bring the just activated window on top again with another SetWindowPos with HWND_TOP. This feels like the wrong way to do it and may cause flicker.

Is it possible to have a window just stop being topmost and placing it below the current foreground window?

Solution

The only automatic method to make a window permanently on top of another one whether the target window is top-most or not is an owner/owned relationship. You could try using SetParent to create this relationship but note that Raymond Chen does say it's not recommended.

Assuming you're tracking window activations somehow, I think your SetWindowPos idea (the first one) is the way to do it, with the following modification:

When the target window is active, set your window to HWND_TOPMOST
When the target loses activation, insert your window after the target window's immediate predecessor in the z-order (i.e. effectively still on top of the target window, but not top-most)

Something like this psuedo-code:

if (foregroundwindow == targetwindow)
   SetWindowPos(my_window, HWND_TOPMOST, ...);
else
{
    HWND hwndPred = GetWindow(targetwindow, GW_HWNDPREV);
    if (!hwndPred)
    {
        // no predecessor so my_window will still be on top, just not top-most any more
        if (GetWindowLong(targetwindow, GWL_EXSTYLE) & WS_EX_TOPMOST)
            hwndPred = HWND_NOTOPMOST;
    }
    SetWindowPos(my_window, hwndPred, ...);
}