How to compute Lucas Kanade flow?

Question

I am currently working on a project of object tracking and have used C++ and OpenCV. I have successfully used Farneback dense optical flow to implement segmentation methods such as k means (using the displacement in each frame). Now I want to do the same thing with Lucas Kanade sparse method. But the output of this function is:

nextPts – output vector of 2D points (with single-precision floating-point coordinates) containing the calculated new positions of input features in the second image; when OPTFLOW_USE_INITIAL_FLOW flag is passed, the vector must have the same size as in the input.

(as stated by the official site)

My question is How I am going to get the result to a Mat flow? for example I have so far tried:

    // Implement Lucas Kanade algorithm
    cvCalcOpticalFlowPyrLK(frame1_1C, frame2_1C, pyramid1, pyramid2,
                           frame1_features, frame2_features, number_of_features,
                           optical_flow_window, 5, optical_flow_found_feature,
                           optical_flow_feature_error, optical_flow_termination_criteria,
                           0);
    // Calculate each feature point's coordinates in every frame
    CvPoint p, q;
    p.x = (int)frame1_features[i].x;
    p.y = (int)frame1_features[i].y;

    q.x = (int)frame2_features[i].x;
    q.y = (int)frame2_features[i].y;
    // Creating the arrows for imshow

    angle      = atan2((double)p.y - q.y, (double)p.x - q.x);
    hypotenuse = sqrt(square(p.y - q.y) + square(p.x - q.x));

    /* Here we lengthen the arrow by a factor of three. */

    q.x = (int)(p.x - 3 * hypotenuse * cos(angle));
    q.y = (int)(p.y - 3 * hypotenuse * sin(angle));

    cvLine(frame1, p, q, line_color, line_thickness, CV_AA, 0);

    p.x = (int)(q.x + 9 * cos(angle + pi / 4));
    p.y = (int)(q.y + 9 * sin(angle + pi / 4));
    cvLine(frame1, p, q, line_color, line_thickness, CV_AA, 0);
    p.x = (int)(q.x + 9 * cos(angle - pi / 4));
    p.y = (int)(q.y + 9 * sin(angle - pi / 4));
    cvLine(frame1, p, q, line_color, line_thickness, CV_AA, 0);

    allocateOnDemand(&framenew, frame_size, IPL_DEPTH_8U, 3);
    cvConvertImage(frame1, framenew, CV_CVTIMG_FLIP);

    cvShowImage("Optical Flow", framenew);

This is the optical flow presentation. Any ideas on how I should get a Mat flow similar to the result of the Farneback optical flow?

(http://docs.opencv.org/2.4/modules/video/doc/motion_analysis_and_object_tracking.html#calcopticalflowfarneback )

UPDATE: Very good answer. But now I have problems with showing the kmeans image. With farneback I used:

    cv::kmeans(m, K, bestLabels,
               TermCriteria(CV_TERMCRIT_EPS + CV_TERMCRIT_ITER, 10, 1.0),
               3, KMEANS_PP_CENTERS, centers);
    int colors[K];
    for (int i = 0; i < K; i++) {
        colors[i] = 255 / (i + 1);
    }

    namedWindow("Kmeans", WINDOW_NORMAL);

    Mat clustered = Mat(flow.rows, flow.cols, CV_32F);

    for (int i = 0; i < flow.cols * flow.rows; i++) {
        clustered.at<float>(i / flow.cols, i % flow.cols) =
            (float)(colors[bestLabels.at<int>(0, i)]);
    }
    clustered.convertTo(clustered, CV_8U);
    imshow("Kmeans", clustered);

Any ideas?

Answer 1

To get an image like the Farneback algorithm you must first understand what is the output.

In OpenCV docs you have:

prev(y,x) ~ next(y + flow(y,x)[1], x +flow(y,x)[0])

So, it is a matrix with the displacements between image 1 and 2. Assuming that the points that you are not calculating will be without movement 0,0; you can simulate this, you only have to put for each of the points (x,y) having the new position (x', y'):

cv::Mat LKFlowMatrix(img.rows, img.cols, CV_32FC2, cv::Scalar(0,0));
LKFlowMatrix.at<cv::Vec2f>(y,x) = cv::Vec2f(x-x', y-y') ;

Also, don't forget to filter the "not found points" with the status = 0

By The Way, your functions are not the opencv c++ version of it:

cvCalcOpticalFlowPyrLK should be cv::calcOpticalFlowFarneback in c++ cvShowImageshould be cv::imshowin c++ and so on

** UPDATE **

Since you need is an input for kmeans (I suppose that is the OpenCV version), and you want to use only the Sparse Points, then you can do something like this:

cv::Mat prevImg, nextImg;
// load your images

std::vector<cv:Point2f> initial_points, new_points;
// fill the initial points vector 

std::vector<uchar> status;
std::vector<float> error;

cv::calcOpticalFlowPyrLK(prevImage, nextImage, initial_points, new_points, status, errors);

std::vector<cv::Vec2f> vectorForKMeans;
for(size_t t = 0; t < status.size(); t++){
  if(status[t] != 0)
    vectorForKmeans.push_back(cv::Vec2f(initial_points[i] - new_points[i]));
}

// Do kmeans to vectorForKMeans