Search code examples
pythonuser-interfaceremote-control

Make your program USE a gui

I'd like to write a program able to "use" other programs by taking control of the mouse/keyboard and being able to "see" what's on the screen.

I used AutoIt to do something similar, but I had to cheat sometimes because the language is not that powerful, or maybe it's just that I suck and I'm not able to do that much with it :P

So... I need to:

  • Take screenshots, then I will compare them to make the program "understand", but it needs to "see"
  • Use the mouse: move, click and release, it's simple, isn't it?
  • Using the keyboard: pressing some keys, or key combinations, including special keys like Alt,Ctrl etc...

How can I do that in python?
Does it works in both linux and windows? (this could be really really cool, but it is not necessary)

Solution

  • I've had some luck with similar tasks using PyWinAuto.

    pywinauto is a set of python modules to automate the Microsoft Windows GUI. At it's simplest it allows you to send mouse and keyboard actions to windows dialogs and controls.

    It also has some support for capturing images of dialogs and such using the Python Imaging Library PIL.