Algorithm for real-time tracking of several simple objects

Question

I'm trying to write a program to track relative position of certain objects while I'm playing the popular game, League of Legends. Specifically, I want to track the x,y screen coordinates of any "minions" currently on the screen (The "minions" are the little guys in the center of the picture with little red and green bars over their heads).

I'm currently using the Java Robot class to send screen captures to my program while I'm playing, and am trying to figure out the best algorithm for locate the minions and track them so long as they stay on the screen.

My current thinking is to use a convolutional neural network to identify and locate the minions by the the colored bars over there heads. However, I'd have to re-identify and locate the minions on every new frame, and this seems like it'd be computationally expensive if I want to do this in real time (~10-60 fps).

These sorts of computer vision algorithms aren't really my specialization, but it seems reasonable that algorithms exist that exploit the fact objects in videos move in a continuous manner (i.e. they don't jump around from frame to frame).

So, is there an easily implementable algorithm for accomplishing this task?

Answer 1

Since this is a computer game, I think that the color of the bars should be constant. That might not be true only if the dynamic illumination affects the health bar, which is highly unlikely.

Thus, just find all of the pixels with this specific colors. Then you do some morphological operations and segment the image into blobs. By selecting only the blobs that fit some criteria, you can find the location of the units.

I know that my answer does not involve video, but the operations should be so simple, that it should be very quick.

As for the tracking, just find per each point the closest in the next frame.

Since the HUD location is constant, there should be no problem removing it.

Here is mine quick and not-so-robust implementation in Matlab that has a few limitations:

1. Units must be quite healthy (At least 40 pixels wide)
2. The bars do not overlap.

---

 function FindUnits()
    x = double(imread('c:\1.jpg'));
    green = cat(3,149,194,151); 

    diff = abs(x - repmat(green,[size(x,1) size(x,2)]));
    diff =  mean(diff,3);
    diff = logical(diff < 30);
    diff = imopen(diff,strel('square',1));

    rp = regionprops(diff,'Centroid','MajorAxisLength','MinorAxisLength','Orientation');
    long = [rp.MajorAxisLength]./[rp.MinorAxisLength];
    rp( long < 20) = [];

    xy = [rp.Centroid];
    x = xy(1:2:end);
    y = xy(2:2:end);
    figure;imshow('c:\1.jpg');hold on ;scatter(x,y,'g');
end

And the results: