C Return Value Conventions (Bool or Err Code)

I get this is probably a loaded question, but in C, is it better practice to return a int with error code, or a "success?" boolean. I notice that "0" for success seems convention in libc, but when working with windows.h, they tend to return "1" on success.

I prefer "0" on success because I can encode different errors, I know others who prefer "1" because of the boolean cast,

Is there any defined/prefered right way, or is it just random?

Solution

It all depends on if you are writing some low-level/generic stuff, in which case you can pretty much return anything including data, or if you are writing a high-level API to be used by others.

In case of the latter, there are indeed best design practices which are widely accepted, that go along the lines of this:

The return value of any public function part of a library/API should be reserved for a result code.
Results should be as detailed as possible meaning enum or equivalent with a number of codes per library/API. The result type is typically declared along with the library header.
Ideally all functions part of that same library/API should use that same result type.
Number zero in the enum is most often used to indicate OK/no error.
Each function documents which results it may return.
Consequently, return types that are just bool, 1/0 or null pointers etc are not recommended for such high level code.
Using some global error variable like errno, GetLastError or whatever it may be named is a big no-no. These are known to be very error prone, users forgetting to clear them etc. And they can potentially cause problems during multi-threading, even when access to the global resource is thread-safe in itself.