Error and status outputs in Lucas-Kanade tracker in OpenCV

Question

Having read the OpenCV documentation, I still can not exactly figure out what st, err are in p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params), and when I need to use them.

Answer 1

It would be better if you were more specific about what you don't understand about them. The documentation for calcOpticalFlowPyrLK() says:

status – output status vector (of unsigned chars); each element of the vector is set to 1 if the flow for the corresponding features has been found, otherwise, it is set to 0.
err – output vector of errors; each element of the vector is set to an error for the corresponding feature, type of the error measure can be set in flags parameter; if the flow wasn’t found then the error is not defined (use the status parameter to find such cases).

So say you have a feature on the border of your image or frame, and in the next image or frame, the feature disappears. The feature would not be found in the next frame, so then the corresponding index for that feature in status would be 0. Otherwise, status would be 1 at that index.

The err value isn't defined wherever status is 0. Otherwise, the error is basically "how well do the neighboring pixels around the feature correspond between the two images/frames". This is a little more clear under the flags param:

OPTFLOW_LK_GET_MIN_EIGENVALS use minimum eigen values as an error measure (see minEigThreshold description); if the flag is not set, then L1 distance between patches around the original and a moved point, divided by number of pixels in a window, is used as a error measure.

The L1 distance is just the sum of absolute differences inside the window. So this just looks in a window around the feature in both images/frames and calculates the absolute difference between the corresponding pixels in each window, and then sums them. So this is what happens when the flag is not set. If it is, the documentation instructs us to look at the description of the minEigThreshold parameter:

minEigThreshold – the algorithm calculates the minimum eigen value of a 2x2 normal matrix of optical flow equations (this matrix is called a spatial gradient matrix in [Bouguet00]), divided by number of pixels in a window; if this value is less than minEigThreshold, then a corresponding feature is filtered out and its flow is not processed, so it allows to remove bad points and get a performance boost.

So with the flag, you can see the minimum eigen values for each feature---useful if you want to figure out a threshold to use. In the original paper for pyramidal implementation of the LK feature tracker referenced in the OpenCV docs (Bouguet00), it details what this means in Section 3 (Feature Selection):

So far, we have described the tracking procedure that takes care of following a point u on an image I to another location v on another image J. However, we have not described means to select the point u on I in the first place. This step is called feature selection. It is very intuitive to approach the problem of feature selection once the mathematical ground for tracking is led out. Indeed, the central step of tracking is the computation of the optical flow vector η^k (see pseudo-code of algorithm in section 2.4). At that step, the G matrix is required to be invertible, or in other words, the minimum eigenvalue of G must be large enough (larger than a threshold). This characterizes pixels that are "easy to track".