I am a newbie to Linux. I have two User space processes, A and B, and B has to receive messages from A, do some processing, and ack when done. So I was looking at a two-way messaging protocol and was seeing netlink being used for communication between user and kernel space. Is there a way to use netlink for two user space process communication?
If it is not an ideal solution, is there any other way to achieve this? I was looking at message queues, but they seem to be one-way communication mechanism.
Netlink was originally designed to provide kernel-userspace communication. There is no reason why it can't be used for userspace-userspace communication, but that said, I don't see why you would.
If you want to go ahead and use it, you can do so solely in userspace. There is no need to perform any of the setup in kernel space first. Just call socket() using a socket family of AF_NETLINK. To send a message, populate a struct sockaddr_nl and set the nl_pid property appropriately (this is generally set to the PID of the current process), then call sendto(). A standard recv() call can be used to receive messages.
All that said, and given that you say you are new to Linux, I would suggest that you look at Unix domain sockets for your userspace IPC needs since I suspect it would satisfy your requirements and should generally be easier to use. You could also look at message queues which can work quite well in some instances. There is a nice comparison here: Which is better for local IPC, POSIX message queues (mqueues) or Unix domain (local) sockets?. Note that you will need to link against the real-time library (librt) in order to use POSIX message queues. Bi-directional communication using message queues can easily be achieved using a pair of queues, one for each direction.