I would like to be able to control from inside my library if it is allowed to be loaded or not without using exceptions, meaning for some cases i would like dlopen("mylib.so") to return NULL and only if all conditions are right it will succeed.
Many have asked about the motivation, inside my library i use dlopen several times and i want to make sure all needed components have been loaded before my library can be loaded.
Take in mind i have to use standard solutions, meaning i can't use external plugins or do things like rewriting dlopen.
As far as I remember, no, you can't, at least in a normal way. If the library exists at given path, it will be loaded. The dlopen it is not designed to do any "business checks" for you at your discretion. At most, it will abide any filesystem permissions/etc and return error if process has no access to the file, and that's it.
If you have full control on the code that will load your library, then wrap it like Atterson suggested and use dlopen2 and it's done.
If you don't have full control, then dlopen2 would still not prevent anyone from using the original dlopen and bypass the checks. You could try to make it more smart, for example, make your dlopen2 do something detectable so then the library can deny working if it was opened by dlopen instead of dlopen2, but then.. someone could fake up that "something detectable", then use dlopen, done. Then it boils down to making that "something detectable" hard to reproduce by attackers.
Simply, it was not intended to do these things. It's meant to load the library if the OS allows (~filesystem permissions, etc).
For any other "access checks" like "do you have license? no? then go away" you have to implement it inside the library. Let them load it via dlopen, then make the library check the permissions for example, at each call to its functions. Do use exceptions, or just do-nothing and return NULLs. Or even better, you probably could use initialization function (see https://stackoverflow.com/a/1602459/717732 + http://tldp.org/HOWTO/Program-Library-HOWTO/miscellaneous.html#INIT-AND-CLEANUP) and do the check once when the lib gets loaded. Note that this functions take and return void so there's still no way to make the dlopen fail, but at least the library can have its moment to disable its functions.