Search code examples
python-3.xwindowspywinauto

How to differentiate two controls on pywinauto

I am trying to create a macro builder using pywinauto. I am developing two applications, the macro recorder and the macro player. The recorder watch every mouse and keyboard events, then save it to an json file, so the the macro player is able to recreate the events.

When the user clicks on an element, the recorder transforms that mouse coordinates UIAWrapper using the following code:

def coords_to_UIAWrapper(coords):
    x, y = coords
    elem = IUIA().iuia.ElementFromPoint(tagPOINT(x, y))
    element = UIAElementInfo(elem)
    wrapper = UIAWrapper(element)
    return wrapper

Then the recorder saves the event and the wrapper.window_text()on a json file, like this:

{
    "events": [
        {
            "device": "mouse",
            "action": "press",
            "text": "<TODOS>"
        },
        {
            "device": "mouse",
            "action": "release",
            "text": "<TODOS>"
        }
    ]
}

And this is an image of the button:

Button

But now the problem happens, because the program has more than one button with the window_text equals <TODOS>:

All buttons

The distinguish the elements I thought I could use the following attributes (inspect.exe): AutomationId, ClassName, RuntimeId, Name, LocalizedControlType, but the LocalizedControlType, ClassNAme and Name are the same for all buttons and AutomationId and RuntimeId are differents everytime I open the application. So, how can I different two elements, to be able to serialize the event and replicate it the macro player.

Solution

  • Interesting work! We're working on script recorder that will be part of pywinauto (visible pull request will be created soon). You can take a look at the architecture here: 'recorder/event_patterns' branch in the fork. I would like to invite you to the discussion soon. Some help would be great. There are still many issues in milestone pywinauto 0.7.0

    The main idea to overcome your issue is keeping the whole UI tree with all parents and children. So full window specification may help to distinguish the buttons by parent. There are other rules in best_match algorithm to separate things: Getting Started Guide -> How to know magic attribute names.