When camera is moving forward/backward, I think stereo rectification would not work. I guess I have to search along the epipolar line directly. Currently, my only clue is to use LineIterator to iterate through epipolar line, and use its neighbors as descriptor. This is very unintelligent method. So I want to know if there is other ways, or helpful opencv functions.
Old problem - Mark Pollefeys had a nice solution for it about 18 years ago. The idea is to rectify in polar coordinates.
See his short paper and lecture notes, and look around for tech reports from there.