Q: KDB+/PyQ publisher subscriber with feeds in python objects

I've been trying to reproduce the publisher subscriber example: https://code.kx.com/wiki/Cookbook/publishsubscribe

Everything's working fine as far as it's purely q related.

I've been trying to reproduce the subscriber in a jupyter notebook using the PyQ kernel. I didn't manage to successfully get the subscriber output/feeds in a python object. Every single time, the subscriber prints in the kernel output (console).

I've also been trying to replicate in PyQ something similar to what I've seen on the java client for q without much success(https://code.kx.com/q/interfaces/java-client-for-q/)

What I would ideally like to achieve is:

running my publisher in command line (so far that was the very easy part)
having a python notebook that listens to the port where my publisher publishes and handles events when a new feed comes in
a websocket solution would be outstanding

Does anyone has some pointers on how to do that on the python side? any examples?

Thanks a lot, Yael Darmon

Solution

In order to get q output redirected to the Jupyter notebook you should specify -1 (for stdout) and -2 (for stderr) options to the %%q cell magic.

I want it in a python object

You can capture standard streams by redirecting them to a pipe. For example, on Linux you can do

>>> import os
>>> r,w = os.pipe()
>>> q('\\2 /dev/fd/%d' % w)
k('::')
>>> q('-2 "hello"')
k('-2')
>>> os.read(r, 5)
b'hello'

Note that I used stderr for illustration because redirecting stdout would interfere with the REPL display, but the same technique would work for stdout. The usual caveats for working with pipes apply. Pipes have a limited buffer, so you need to arrange regular reads. Typically this will involve some kind of event management which is outside of the scope of this answer. If you don't want to deal with such complexity, your best bet is to use a temporary file instead of a pipe. I would also recommend studying how "fdcap" fixture implemented in pytest.