I need to compare every frame with the previous and determine in which direction the movement has occurred. Is this possible by just getting the difference between the two frames? Will it work with slow motion of about 2 m/s? Also, how could I calculate direction of motion using the same?
You should read about optical flow analysis.