I am using MATLAB's PsychToolbox to run an experiment where I have to gather saccade information in real time, while also rendering a video frame by frame. The problem that I have is that given the frame rate of the video and the display (~24fps), it means that I have about 40ms time window to render query and render every frame that I have previously stored in memory. This is fine, but since this process takes aditional time, it usually implies that I have about ~20ms to consistently poll for a saccade from beginning to end.
This is a problem, because when I poll for saccades, what I am usually doing (in say still images, that only have to be displayed once), is I wait for a start and end of a fixation, given consistent polling from the eye tracking machine, that detects that the observers gaze has shifted abruptly from one point to another with a
speed exceeding: 35 deg/s
and an
acceleration exceeding: 9500 deg/s^2
but if the beginning of a saccade or end of it takes places when a frame is being rendered (Which is most of the time), then it makes it impossible to get the data in real time without splitting the rendering and polling process into two separate MATLAB threads.
My code (relevant part) looks like this:
while GetSecs-t.stimstart(sess,tc)<fixation_time(stimshownorder(tc))
x =evt.gx(1);
y =evt.gy(1);
pa = evt.pa(1);
x_vec = [x_vec; x];
y_vec = [y_vec; y];
pa_vec = [pa_vec; pa];
evta=Eyelink('NewestFloatSample');
evtype=Eyelink('GetNextDataType');
#%% Ideally this block should detect saccades
#%% It works perfect in still images but it can't do anything here
#%% since it conflicts the main for loop ahead.
if evtype==el.ENDSACC
sacdata=Eyelink('GetFloatData',evtype);
sac.startx(sess,tc,sacc)=sacdata.gstx;
sac.starty(sess,tc,sacc)=sacdata.gsty;
sac.endx(sess,tc,sacc)=sacdata.genx;
sac.endy(sess,tc,sacc)=sacdata.geny;
sac.start(sess,tc,sacc)=sacdata.sttime;
sac.end(sess,tc,sacc)=sacdata.entime;
sacc=sacc+1;
end
#%Main loop where we render each frame:
if (GetSecs-t.space(sess,tc)>lag(tc))
z = floor((GetSecs-t.space(sess,tc)-lag(tc))/(1/24))+1;
if z > frame_number
z = frame_number;
end
Screen('DrawTexture',win,stimTex{z});
Screen('Flip',win);
#DEBUG:
#disp(z);
#%disp(frame_number);
end
end
Ideally, I'd want a MATLAB function that can render the video independently in one separate thread in the back end, while still polling for saccades in the main thread. Ideally like this:
#% Define New thread to render video
#% Some new function that renders video in parallel in another thread
StartParallelThread(1);
#%Play video:
Playmovie(stimTex);
#%Now start this main loop to poll for eye movements.
while GetSecs-t.stimstart(sess,tc)<fixation_time(stimshownorder(tc))
x =evt.gx(1);
y =evt.gy(1);
pa = evt.pa(1);
x_vec = [x_vec; x];
y_vec = [y_vec; y];
pa_vec = [pa_vec; pa];
evta=Eyelink('NewestFloatSample');
evtype=Eyelink('GetNextDataType');
if evtype==el.ENDSACC
sacdata=Eyelink('GetFloatData',evtype);
sac.startx(sess,tc,sacc)=sacdata.gstx;
sac.starty(sess,tc,sacc)=sacdata.gsty;
sac.endx(sess,tc,sacc)=sacdata.genx;
sac.endy(sess,tc,sacc)=sacdata.geny;
sac.start(sess,tc,sacc)=sacdata.sttime;
sac.end(sess,tc,sacc)=sacdata.entime;
sacc=sacc+1;
end
end
It also seems that the time it takes to run the Screen('Flip',win) command is about 16ms. This means that if any saccades happen in this interval, I would not be able to detect or poll them. Note that in the end I am having 42ms (for frame refresh rate) minus 16ms (for time it takes to query and display the frame), so a total of ~26ms of probing time per frame for getting eye movements and computing any real-time processing.
A possible solution might be to continually poll for gaze, instead of checking if an eye movement is a saccade or not. But I'd still have the problem of not capturing what goes on in about a third of each frame, just because it takes time to load it.
You need to reorganize your code. The only way to make this work is knowing how long the flip takes and knowing how long submission of the next video frame takes. Then you poll the eye tracker in a loop until you have just enough time left for the drawing commands to be executed before the next screen vertical blank.
You can't do any form of reliable multi-threading in matlab