Capturing Display / Monitor Images, Sending Keyboard Input on Linux

Question

I need to process images sent to my laptop's video display, and I need to send keyboard input to my Linux system, using a C++ or shell program.

My goal is to process images that are part of an FPS game, then taking action inside that game (hence the keyboard input) based on these images. Instead of trying to understand (if it's even possible) how to interface to game X or Y, using some API, I figured this is the quickest way to interface to any game, hijacking Linux input and output somehow.

Is there any way to do this without any kernel, or device driver hacking? I used recordmydesktop to record my desktop as a video before, I guess I could hack its code and try to reverse-engineer something from that. Any other ideas? I am on Ubuntu 11.

Related Question

Answer 1

I finally have a solution. I believe UrT loads OpenGL on its own so that things such as wallhacks, etc are not possible. Then the best remaining option is taking X screenshots. This worked quite fast, even from a scripting language like Python. The following code takes successive screenshots and displays them as an animation through OpenCV. You need to start UrT in minimized mode of course. The rest of the details are in my project.

import gtk.gdk
import PIL
from opencv.cv import *
from opencv.highgui import *
from opencv.adaptors import PIL2Ipl

w = gtk.gdk.get_default_root_window()
sz = w.get_size()
print "The size of the window is %d x %d" % sz

size_x = 600
size_y = 400
start_x = 0
start_y = 100
end_x = start_x+size_x
end_y = start_y+size_y
box = (start_x, start_y, start_x+size_x, start_y+size_y)

while True:
    pb = gtk.gdk.Pixbuf(gtk.gdk.COLORSPACE_RGB,False,8,sz[0],sz[1])
    pb = pb.get_from_drawable(w,w.get_colormap(),0,0,0,0,sz[0],sz[1])
    width,height = pb.get_width(),pb.get_height()
    im = PIL.Image.fromstring("RGB",(width,height),pb.get_pixels())
    im = im.crop(box)
    cv_img = PIL2Ipl(im)
    cvNamedWindow("fps")
    cvShowImage("fps", cv_img)
    cvWaitKey(30) 

Also, for sending keys to the game, the method above did not work, I had to use xdotool in order to send forward walk key to UrT,

xdotool search --name ioUrbanTerror  windowactivate keydown W