I get this is probably a loaded question, but in C, is it better practice to return a int with error code, or a "success?" boolean. I notice that "0" for success seems convention in libc, but when working with windows.h, they tend to return "1" on success.
I prefer "0" on success because I can encode different errors, I know others who prefer "1" because of the boolean cast,
Is there any defined/prefered right way, or is it just random?
Is there any defined/preferred right way
No
or is it just random?
No.
0 for success appears whenever nonzero for success would not suffice. Cases:
memcmp() and friends are really three-way comparison functions but in practice are almost always not equal/equal. Some binary comparison functions picked this behavior up: 0 for equal, 1 for nonequal.
Processes return 0 for success and nonzero (most platforms restrict from 1 to 255) for error code. This has carried over to functions that start processes and wait for them to return.
Windows COM functions and Windows Native functions return high bit clear for success and high bit set for error. In most cases there is only one valid success status which is 0 so people test for it directly (treating an unexpected success status as an error, as they probably should).
Unix system calls return -errno on failure (including functions such as mmap that usually return pointers), and libc stuffs the error in errno and returns -1. In old hand Unix code you see < 0 as tests for error for reasons including this one. Some Unix library functions inherited either the direct system call approach or the libc-system call approach either because they were low level right next to libc-system calls or by picking up the thing as though it were a platform preference.
Most other things resulted in either 0 for failure and 1 for success or 0 for failure and any nonzero for success (a bunch of Windows functions declared to return BOOL did this on older Windows NT versions).
There's enough 0 for success cases lying around that it probably isn't too bad if you used 0 for success somewhere for no good reason. Somebody may well pick a bone with you, but that someone won't be me.
Personal commentary: Question got closed opinion based after I posted this answer. I wrote it to be as non-opinion based as possible assuming the question actually was on topic, and as such there is very little opinion here.