Search code examples
javaswingfocuscontextmenuawtrobot

How can I control context menus in other (Windows) applications using Java Robot and Swing buttons


I have written an extensive on-screen keyboard (OSK) in Java. This is intended for people with disabilities and needs to be able to do everything that a standard keyboard can do. One thing I've noticed that doesn't work is context menus - in (most) Windows programs. I want my OSK user to be able to click the 'Context Menu' key (normally beside right-Ctrl on standard keyboard) and use the arrow keys to navigate that menu. The context menu appears fine, but when I click back onto my Frame (to press an arrow key), the context menu disappears.

Anybody got any ideas as to how this might be achieved (probably a fairly monstrous 'hack' required I would think)? I'm pretty sure the context is disappearing because a mouse-click is occurring elsewhere on the screen (hard to get around that when you need to actually click something!).

Below is a test app. I can use it to navigate down through a context menu in NetBeans successfully, but not in any other application I've tried (e.g. Windows Explorer).

import java.awt.Robot;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.event.KeyEvent;
import javax.swing.JButton;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.WindowConstants;

public class RobotContextMenuTest extends JFrame implements ActionListener
{
    Robot R;
    JPanel JP;
    JButton JBmenu, JBdown;

    public RobotContextMenuTest()
    {
        try
        {
            R = new Robot();
        }
        catch(Exception e)
        {
            e.printStackTrace();
        }

        JP = new JPanel();
        getContentPane().add(JP);

        JBmenu = new JButton("Menu");
        JBmenu.addActionListener(this);
        JP.add(JBmenu);

        JBdown = new JButton("Down");
        JBdown.addActionListener(this);
        JP.add(JBdown);

        setFocusableWindowState(false);
        setAlwaysOnTop(true);
        setSize(200,80);
        setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
        setVisible(true);
    }
    public static void main(String args[])
    {
        new RobotContextMenuTest();
    }

    public void actionPerformed(ActionEvent e)
    {
        if(e.getActionCommand().equals("Menu"))
        {
            R.keyPress(KeyEvent.VK_CONTEXT_MENU);
            R.keyRelease(KeyEvent.VK_CONTEXT_MENU);
        }
        else if(e.getActionCommand().equals("Down"))
        {
            R.keyPress(KeyEvent.VK_DOWN);
            R.keyRelease(KeyEvent.VK_DOWN);
        }
    }
}

Finally, here's what I've tried:

  • Using MouseListener on the buttons instead of ActionListener, then 'consuming' the MouseEvent in the hope that the other application won't register it and hide the context menu.
  • I made a Threaded test application that is able to show any context menu and automatically scan down through it (once every second) but that doesn't really solve my problem! I want the context menu navigation to be controlled by the user's mouse clicks.
  • I've tried using JNativeHook as an alternative method of capturing mouse clicks but the result is the same.

Sorry for the long-winded question but it's a fairly complicated problem! Incidentally, the standard Windows 7 OSK can't do this either but WiViK can. Thanks.


Solution

  • Finally cracked this with a lot of help from a friend (thanks Chris Gregan!). After many failed attempts, the answer was relatively simple. I used JNA and Windows API to create a hook for low-level mouse events. It's all described with 2 sample classes given here in an answer by 'prunge'. Note: I had to include the jars for jna version 3.4.0 and jna platform version 3.4.0 (newer versions of these jars resulted in errors!).

    Using the above linked MouseHook, you can prevent a low-level event from propagating (reaching other applications) by returning -1 from the LowLevelMouseProc callback.

    Here is a snippet I added to the callback method in MouseHook to intercept/suppress left mouse button presses (thereby preventing context menus from disappearing!):

    if(nCode >= 0 && wParam.intValue() == WinUserX.WM_LBUTTONDOWN)
        return new LRESULT(-1);
    

    Thanks again to Alex Barker for taking the time to look at this and point me in the direction of JNA and Windows API.