Search code examples
pythonpyautogui

PyAutoGUI - result will only get parsed once


I'm trying to automate a simple thing. A popup pops up, my script finds it and clicks on the button I took a screenshot of (erteilen.png). Half a second later the same popup pops up again, but my script won't notice it.

This is my code:

import pyautogui as pg

while True:
    window = pg.locateOnScreen("erteilen.png", minSearchTime=3, confidence=0.9)
    try:
        print(window)
        pg.click(x=window.left+int(window.width/2), y=window.top+int(window.height/2), clicks=3)
    except Exception as e:
        print("Error:", e)
        break

It will click on the found window, then the same window opens again, but the script won't locate it again. When I click in the command window, it seems like pyautogui refreshes it's screen and then finds the window.

How can I tell PyAutoGUI to 'refresh' what it sees itself, so that it gets updated informations and finds the new windows as well?


Solution

  • There is no such a function which would "refresh" the screen, which Pyautogui uses. It takes copy of the screen and analyse it every pg.locateOnScreen call. You can confirm, that script is doing it's work based on resources used. You can see that CPU load is high, when script is working.

    I was trying to reproduce Your issue on Windows 10 and Linux. Pyautogui was working much more reliable on Linux than on Windows. For the same searched button I had to lower confidence to 0.6 on Windows to make same things work. On Linux confidence 0.9 was working just fine.

    • First thing, that prevent everything for working reliably was Your code itself. When minSearchTime expires, locateOnScreen method returns None. That was causing Your script to crash.

    • You might increase performance by adding grayscale=True in locateOnScreen method call.

    • Also pg.locateOnScreen can raise ImageNotFoundException if image was not found on the screen.

    • Some browsers might involve some strategies preventing robots. For example I could not close popup generated by Brave browser.
      To overcome this issue, You should introduce random timeout between button found in the image and mouse click. Also, You always click on the same spot. You should randomize mouse position a bit.

    I put together code snippet, which was working reliably in Windows and Linux environment closing generated alert message every 3 seconds.

    import random
    from time import sleep
    
    import pyautogui as pg
    from pyautogui import ImageNotFoundException
    
    offset = 3
    while True:
        # Locate button on screen in grayscale mode
        try:
            window = pg.locateOnScreen("search.png", minSearchTime=3, confidence=0.9, grayscale=True)
        except ImageNotFoundException as e:
            # Thrown by pg.locateOnScreen
            # Continue searching
            continue
    
        if window is None:
            # Button was not found in the screen
            # Continue searching
            continue
        # Random sleep from 100 ms to 1 second
        sleep(random.randint(1, 10) / 10)
        # Random mouse position within button
        rand_x_pos = random.randint(int(window.left) + offset, window.left + int(window.width) - offset)
        rand_y_pos = random.randint(int(window.top) + offset, window.top + int(window.height) - offset)
        # Click on the button
        pg.click(x=rand_x_pos, y=rand_y_pos, clicks=3)
    

    This is javascript to generate alert message:

    <!DOCTYPE html>
    <html>
    <head>
    <title>Page Title</title>
    </head>
    <body>
    
    <script type="text/javascript">
      setInterval(function() {
          alert("Message to alert every 3 seconds");
      }, 3000);
    </script>
    
    </body>
    </html>
    

    This is search.png image, which is OK button used in Brave browser.
    enter image description here