I've downloaded a file (ListFunc.mexw64) containing some functions I'd like to use in my algorithm. I am using Mac OS but, it seems incompatible with OS as well as the MATLAB software itself.
Is there a way to get it to work or to see the content of the file?
A mexw64 file is compiled for Win64 systems, the equivalent extension for macOS is mexmaci64 - this file simply wasn't meant to be used on a macOS.
You should try to contact the authors of the file and ask them to recompile it for Mac, or ask for the source code and compile it yourself, according to these instructions.