I'm trying to automate a simple thing. A popup pops up, my script finds it and clicks on the button I took a screenshot of (erteilen.png). Half a second later the same popup pops up again, but my script won't notice it.
This is my code:
import pyautogui as pg
while True:
window = pg.locateOnScreen("erteilen.png", minSearchTime=3, confidence=0.9)
try:
print(window)
pg.click(x=window.left+int(window.width/2), y=window.top+int(window.height/2), clicks=3)
except Exception as e:
print("Error:", e)
break
It will click on the found window, then the same window opens again, but the script won't locate it again. When I click in the command window, it seems like pyautogui refreshes it's screen and then finds the window.
How can I tell PyAutoGUI to 'refresh' what it sees itself, so that it gets updated informations and finds the new windows as well?
There is no such a function which would "refresh" the screen, which Pyautogui uses. It takes copy of the screen and analyse it every pg.locateOnScreen
call. You can confirm, that script is doing it's work based on resources used. You can see that CPU load is high, when script is working.
I was trying to reproduce Your issue on Windows 10 and Linux. Pyautogui was working much more reliable on Linux than on Windows. For the same searched button I had to lower confidence to 0.6 on Windows to make same things work. On Linux confidence 0.9 was working just fine.
First thing, that prevent everything for working reliably was Your code itself. When minSearchTime
expires, locateOnScreen
method returns None
. That was causing Your script to crash.
You might increase performance by adding grayscale=True
in locateOnScreen
method call.
Also pg.locateOnScreen
can raise ImageNotFoundException
if image was not found on the screen.
Some browsers might involve some strategies preventing robots. For example I could not close popup generated by Brave browser.
To overcome this issue, You should introduce random timeout between button found in the image and mouse click. Also, You always click on the same spot. You should randomize mouse position a bit.
I put together code snippet, which was working reliably in Windows and Linux environment closing generated alert
message every 3 seconds.
import random
from time import sleep
import pyautogui as pg
from pyautogui import ImageNotFoundException
offset = 3
while True:
# Locate button on screen in grayscale mode
try:
window = pg.locateOnScreen("search.png", minSearchTime=3, confidence=0.9, grayscale=True)
except ImageNotFoundException as e:
# Thrown by pg.locateOnScreen
# Continue searching
continue
if window is None:
# Button was not found in the screen
# Continue searching
continue
# Random sleep from 100 ms to 1 second
sleep(random.randint(1, 10) / 10)
# Random mouse position within button
rand_x_pos = random.randint(int(window.left) + offset, window.left + int(window.width) - offset)
rand_y_pos = random.randint(int(window.top) + offset, window.top + int(window.height) - offset)
# Click on the button
pg.click(x=rand_x_pos, y=rand_y_pos, clicks=3)
This is javascript to generate alert message:
<!DOCTYPE html>
<html>
<head>
<title>Page Title</title>
</head>
<body>
<script type="text/javascript">
setInterval(function() {
alert("Message to alert every 3 seconds");
}, 3000);
</script>
</body>
</html>
This is search.png
image, which is OK
button used in Brave browser.