How to know if a TToolButton's MenuItem is dropped down?

In the context of an owner-draw toolbar used to host menu entries (TToolButtons with their MenuItem and Grouped properties set), I want to know if the corresponding menuitem is dropped. The problem is that the State property in the OnAdvancedCustomDrawButton doesn't reflect that information.

When the toolbutton is clicked, its Down property is true, but in the particular case above (MenuItem set and Grouped=True), just after the menu is dropped, another OnAdvancedCustomDrawButton is fired but this time with Down set to false.

This means that I end up drawing the button with a NOT down state.

Looking at the source of the VCL, it seems the information about which toolbutton is dropped is stored in the TToolBar's FMenuButton private field, and Windows is notified of the hot state by a Perform(TB_SETHOTITEM), however neither of these provide read-access...

Also the VCL performs the dropdown via a private FTempMenu, whose handle is thus not accessible.

PS: FWIW if using the hacky solution, the only private field usable seems to be FButtonMenu which will have to be compared against your Button.MenuItem in the CustomDraw, the others private fiels are either not set early enough (like FMenuButton) or are private variables (like MenuButtonIndex) with a variable location. Still not too satisfying though.

Solution

Getting the menu-dropped-down status is problematic, the code that makes the menu pop up is pretty convoluted, makes use of some message hooks. It's generally not the code you'd want to touch. Fortunately the Toolbar itself keeps track of the drop-down menu status, using the FMenuDropped variable. Unfortunately that variable is private, you can't access it from outside, the "hacked" trick doesn't work. Being private it also doesn't offer RTTI!

There are two possible solutions:

Modify the VCL and add a property that makes FMenuDropped available from outside

Go to ComCtrls.pas, find the TToolBar = class(TToolWindow) declaration, go to the public section and add this:

property MenuDropped:Boolean read FMenuDropped;

From your code you'll then be able to check the toolbar if it has a dropped down menu or not. The unfortunate part of this is that it requires modifications to the VCL. Never a good idea, difficult to synchronize amongst several programmers.

Use a hack to access the FMenuDropped field directly, without changing the VCL

To do this you need to get the offset of the FMenuDropped field. Once you get that you can write something like this:

if PBoolean(Integer(Toolbar1) + 865)^ then
   DoStuffIfMenuIsDropped
else
   OtherStuffIfMenuIsNotDropped;

The 865 is actually the correct constant for Delphi 2010! Here's a very quick way of getting the constant.

Go to compiler settings, check "compile using debug DCU's"
Open ComCtrls.pas, go to procedure TToolButton.Paint, place a brakepoint in there.
Start the application, take a piece of paper and a pen. When the program stops at the brakepoint open up Debug Inspector. To do that simply place the cursor on the name of a field, any field, and hit Alt+F5. With the Debug Inspector window hit Ctrl+N to show the generic Inspect editor that allows you to inspect anything. Enter Integer(FToolbar). Note the result on the piece of paper.
Hit Ctrl+N again, this time enter Integer(@FToolBar.FMenuDropped). Note this second number.
The constant you need is the difference between the second and the first. That's it!

There are of course some possible problems. First of all this depends on the exact Delphi version you're using. If the code needs to be compiled on different versions of the Delphi compiler, clever $IFDEF need to be used. None the less this is workable.

(Edit): You can use this same technique to access any Private field of any class. But you'll need to think many times before doing this, because private fields are made private for a reason.