The signature of unlocked_ioctl inside struct file_operations in is
long (*unlocked_ioctl) (struct file *, unsigned int, unsigned long);
while the man 2 ioctl says the signature of ioctl(2) is:
int ioctl(int d, int request, ...);
I know how the parameters get mangled inside the kernel, but why return type in kernel space is long, while the user space gets int? This creates a problem when I want to return a negative value as an error: because of a two-complement encoding, everything negative value I return is turned into -1.
If you return a negative value from a file_operations function, the kernel interprets it as a negative errno (i.e. an error return). User code then gets -1 as a return value, with errno set to the negation of your original return value. This has nothing to do with twos-complement.
Per man 2 intro, “introduction to system calls”:
On error, most system calls return a negative error number (i.e., the negated value of one of the constants described in errno(3)). The C library wrapper hides this detail from the caller: when a system call returns a negative value, the wrapper copies the absolute value into the errno variable, and returns -1 as the return value of the wrapper.
As an example, if you return -ENOTTY from unlocked_ioctl, the user program gets -1 from ioctl and errno = ENOTTY.