I use Octave to run the selective search code, it gives me this error:
error: @SSSimColourTextureSizeFillOrig: no function and no method found.
The code which arises this error is:
simFunctionHandles = {@SSSimColourTextureSizeFillOrig, @SSSimTextureSizeFill, @SSSimBoxFillOrig, @SSSimSize};
But this code can successfully run on MATLAB, so I supposed there is some bugs in Octave or it doesn't support function handle?
Octave has absolutely no problems with function handles. What the exact problem in your case is, is the format of the files.
If you look at the Github repo that is linked in your post, the functions you are referring to are MATLAB P-Code files. These are obfuscated MATLAB files which can only be run in MATLAB. The purpose of P-Code files is to take the original source code and sort of hide what the original source was so you can't take the original source and modify it for your own use. That being said, you have no choice but to use this function for your purposes and you can't modify its behaviour.
However, Octave does not have this kind of obfuscation support and so naturally it can't reconstruct what the original code was for you to run.
Check out this Octave thread for more details: http://octave.1599824.n4.nabble.com/Matlab-p-files-td1627334.html
.... so basically, you're SOL. You can only run those functions in MATLAB. However, you may have some luck by contacting the original author and obtaining the original .m files, but the fact that these are P-Code files is probably because there are some methods and techniques that the author doesn't want to share, but needs to distribute this code in such a way that the rest of the system will work.