Steps to attempt to repeat this error.
./configure
- You can use any options, the error is present with all.sudo make
- It will compile all of the code but when it comes time to link the code it runs into the error
/usr/local/lib/libavcodec.a(libx264.o): In function 'X264_init':
/.../libx264.c:418: undefined reference to 'x264_encoder_open_128'
This error also shows up when building avconv. It also fails at linking time with the same error. So I can assume it is nothing wrong with that directory or source.
That is error of linking with libavcodec.a which depends on old version of libx264. If you need only libx264 than you can compile it without lavf support (./configure --disable-lavf). If you need x264 with lavf support than you need to recompile libavcodec.a without libx264. i.e. you need to break circular dependencies between libavcodec and libx264.