Everything is in the title really. The man page does not say anything. Some google reports some people checking errno after dlopen, but just few and with various results. The function does return a success/non-success value, but how should one figure out the cause of the failure.
So does dlopen() set errno? where are the different errno values defined, then?
If not, what should be used to check what caused the failure?
No (emphasis added).
Upon successful completion, dlopen() shall return a symbol table handle. If file cannot be found, cannot be opened for reading, is not of an appropriate executable object file format for processing by dlopen(), or if an error occurs during the process of loading file or relocating its symbolic references, dlopen() shall return a null pointer. More detailed diagnostic information shall be available through dlerror().
As noted by others, nothing prohibits dlopen() (or, more likely, functions that dlopen() calls) from modifying errno. But a standard-conforming dlopen() will not report its own errors through errno. You need to call dlerror().