I am a newbie to computer vision and while going through the optical flow topic I had a conceptual doubt. Why cant optical flow be calculated by taking difference between x and y values of the pixels in two consecutive frames and dividing them by time between two frames? For example, pixel in image moves from (0,0) to (1,1) in 2 consecutive frame which can be matched using brightness constancy assumption. And by knowing the frame rate one can find out u and v.
Mathematically you can see that it's underconstrained by looking at the Estimation section in http://en.m.wikipedia.org/wiki/Optical_flow
This is referred to as the aperture problem. You can google on this for more discussion, or see http://en.m.wikipedia.org/wiki/Motion_perception#The_aperture_problem for an animated example.