I download some mexw64. type files with some Matlab .m files. I want to ask how can I open them and investigate the code and algorithm through them? Is it possible to convert them to the .m file? (the windows in my laptop is 64 bit)
Unfortunately not, unless you can contact the author of the MEX file.
MEX files are Matlab executable files. A MEX file is first written in C/C++/Fortran, and then compiled to a binary file with extension .mexw64 (Windows), .mexmaci64 (MacOS), or .mexa64 (Linux). The extension is chosen by the author, based on whether he/she intends the file to be used on Windows, Mac, or Linux.
See the documentation for more information, and the Matlab Q&A site for possible indirect solutions.