I've written a 64-bit kernel module that works fine for 64-bit apps, but crashes for 32-bit apps. I need to figure out how how to make the module detect if a client is a 32-bit app, and, if so manage the conversion between the 32-bit struct and 64-bit struct in my module so the system calls work.
When the module is used by a 32-bit app, the recvmsg() function crashes if the msg has cmsghdrs, because the cmsghdr struct has different size in a 32-bit app.
My module doesn't provide a 32-bit compatibility layer explicitly, even though the kernel is configured to support 32-bit applications in compatibility mode.
The module implements a new protocol, defines struct proto_ops, but I don't see compat_recv in the proto ops.
The module acts as a proxy between the client and another application (64-bit), wherein it fetches responses to client requests from the backend using netlink messages as opposed to making __sys_socket*() calls.
compat_recv is not needed in proto_ops. 32 bit userspace call to recvmsg in 64 bit kernel should go through compat_sys_recvmsg, which set MSG_CMSG_COMPAT flag. That flag makes ___sys_recvmsg() do necessary conversions before doing callback to your proto_ops and before returning to userspace.
References (in kernel tree):
Regards, Mateusz.