It's a very strange problem. I think it must caused my incorrect configuration of ansi-term, but i still can't find out where it is.
The issue is: when i in ansi-term and press M-x find-file, the prompt isn't current directory but the path i entered in my previous find file action. So when i change directory, it still display the same directory. So i have to enter the current directory every time. But it works very well in M-x shell and M-x eshell
Does the same thing happen when you start Emacs without your init file, i.e., emacs -Q? If so, that's the designed behavior or (especially if you use a development snapshot) perhaps an Emacs bug.
If not, then bisect your init file recursively to find out which part of it causes this behavior. To do that, use, e.g., command comment-region (see prefix arg in doc) to comment and uncomment a block of text. Comment out 1/2 of your init file, then 3/4, then 7/8,...,
each time testing whether the uncommented portion causes or removes the problematic behavior. You will very quickly identify what causes the behavior.