I am using the keyboard and mouse modules to record user interaction with our software so that we can have some high level GUI tests. Currently I am trying to store the recorded events in a text file and later play this recording again.
However when I load the recorded events from said file I only see played mouse events and no keyboard events.
One cause for this problem may be the implementation of KeyboardEvents.
KeyboardEvents does not contain a correct implementation of __repr__. This prevents us from calling print(keyboard_events, file=f) and reading the lines with eval(line). (This works with mouse and ButtonEvent and MoveEvent)
So we have decided to work with the json format of KeyboardEvents. Basically what we're doing is we retrieve the json format of each KeyboardEvent and write the json in the file. Then we load the json file and parse the json as KeyboardEvents.
Currently we're storing both mouse & keyboard input in a single file. However, since mouse supports a correct implementation of __repr__ we can directly print and the mouse events and use eval() on it to retrieve the stored events.
This is the file used for recording and playing:
import threading
import mouse
import keyboard
from mouse import ButtonEvent
from mouse import MoveEvent
from mouse import WheelEvent
from keyboard import KeyboardEvent
import time
import json
import sys
def record(file='record.txt'):
f = open(file, 'w+')
mouse_events = []
keyboard_events = []
keyboard.start_recording()
starttime = time.time()
mouse.hook(mouse_events.append)
keyboard.wait('esc')
keyboard_events = keyboard.stop_recording()
mouse.unhook(mouse_events.append)
#first line = start of recording
#mouse events = second line
#keyboard events = every remaining line = 1 event
print(starttime, file=f)
print(mouse_events, file=f)
for kevent in range(0, len(keyboard_events)):
print(keyboard_events[kevent].to_json(), file = f)
f.close()
def play(file, speed = 0.5):
f = open(file, 'r')
#per definition the first line is mouse events and the rest is keyboard events
lines = f.readlines()
f.close()
mouse_events = eval(lines[1])
keyboard_events = []
for index in range(2,len(lines)):
keyboard_events.append(keyboard.KeyboardEvent(**json.loads(lines[index])))
starttime = float(lines[0])
keyboard_time_interval = keyboard_events[0].time - starttime
keyboard_time_interval /= speed
mouse_time_interval = mouse_events[0].time - starttime
mouse_time_interval /= speed
print(keyboard_time_interval)
print(mouse_time_interval)
#Keyboard threadings:
k_thread = threading.Thread(target = lambda : time.sleep(keyboard_time_interval) == keyboard.play(keyboard_events, speed_factor=speed) )
#Mouse threadings:
m_thread = threading.Thread(target = lambda : time.sleep(mouse_time_interval) == mouse.play(mouse_events, speed_factor=speed))
#start threads
m_thread.start()
k_thread.start()
#waiting for both threadings to be completed
k_thread.join()
m_thread.join()
if __name__ == '__main__':
if len(sys.argv) > 2 and sys.argv[1] == 'play':
play(sys.argv[2])
elif len(sys.argv) >= 2 and sys.argv[1] == 'record':
if(len(sys.argv)) == 3:
record(sys.argv[2])
else:
record()
else:
print("missing either 'play' or 'record' or filename")
I expect the same behavior with this code like when it is run in a single function (see edit in https://stackoverflow.com/a/57670484/7345513).
Meaning: I expect the playback in the threads to be synced and the keys to be pressed. What i actually get is that the mouse events are played back as desired but no KeyboardEvents are being processed. When I use the function from the linked SO it works.
Can someone please point me to the right direction?
I forgot to answer my own question but here it goes:
Thanks to the guidance from blubberdiblub I have tracked the start time for the tracking and can then add an offset to when the first input occurred. This allows playbacks to be timed somewhat accurately.