I have a Linux server which processes Cap'n Proto RPC requests. Some of those requests need to forward data in the request to another server running, in this case, a Kafka broker. The librdkafka and Cap'n Proto KJ libraries can both use poll() so i think the OS will ensure they can both run asynchronously but I'm not sure if further integration is required or beneficial. Does anyone have experience with this?
The question is a bit broader than the specifics i've listed.. there are other APIs which i may call in future from Cap'n Proto RPC so any broad guidelines would be appreciated.
Unfortunately, this is not so simple. Yes, they both use poll(), but the problem is, only one library will be calling poll() at a time, and only that library will actually receive any events -- the other is stuck. This is a classic challenge with event loop libraries -- by default they cannot be used together.
One option is to try to use the libraries in separate threads. But, often event-driven libraries design around the assumption that you're doing everything in a single thread because otherwise why would you need an event loop?
But "The Right Thing" is to integrate the event loops. KJ's event loop is capable of being integrated with other event libraries. For example, I integrated it with libuv for node-capnp; see the first part of this file:
https://github.com/kentonv/node-capnp/blob/master/src/node-capnp/capnp.cc
(At some point I plan to separate out the libuv-related code here into a separate library shipped with Cap'n Proto.)
For another example, here's a pull request by Nathan Hourt to add integration with Qt's event loop -- but note that this one doesn't include I/O integration, I think because Nathan is using an implementation of AsyncIoStream into which he manually pushes data when it's available:
https://github.com/sandstorm-io/capnproto/pull/253
In any case, you'll need to do something similar with whatever Kafka uses. Hopefully you'll then contribute your code back to Cap'n Proto! :)