Given a file, I want to check if this is a DLL, or a shared object (Linux) or a dylib (Mac OS X), or something different. My main interest is differentiating executable and DLL on Linux and Mac OS X. For windows, the extension should be enough for my problem.
I already checked that the magic number technique doesn't work for Linux as executable and shared objects both have the same number.
The Unix (Linux and Mac OS X) command file knows how to identify file types.
man file tells you about the 'magic' information used to do this, and in particular where the file with that information is to be found.
man 5 magic describes the file in detail and should also tell you where it lives. You can take a look in there and pull anything you need from it. And/or just crib code from the source of file.
Update:
Note that on Linux a file could be both an executable and a shared library at the same time.
One example is /lib/libc.so.6 (a shared library which can be executed).
Another example: any executable built with -pie flag can be used as a shared library. See this answer for details.