Search code examples
pythonpython-3.xnumpykdbpyq

Kdb database to NumPy array in PyQ


I have a splayed Kdb database of symbols, floats, and timestamps. I'd like to convert this to NumPy arrays. However using the following code...

>>> import numpy as np
>>> from pyq import q
>>> d = q.load(':alpha/HDB/')
>>> a = np.array(d)

Returns this error...

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/marrowgari/py3/lib/python3.6/site-packages/pyq/_n.py", line 158, in array
return numpy.array(list(self), dtype)
TypeError: iteration over a K scalar, t=-11

Is this because Kdb symbol types do not have a direct analogue in NumPy? If so, how do I correct this?


Solution

  • Suppose your HDB was created as follows:

    q)(` sv db,`t`)set .Q.en[db:`:alpha/HDB]([]sym:`A`B`C;a:1 2 3)
    `:alpha/HDB/t/
    q)\l alpha/HDB
    q)t
    sym a
    -----
    A   1
    B   2
    C   3
    

    Then, first of all you should load it using \l command, not the load function:

    >>> q('\\l alpha/HDB')
    k('::')
    

    This will load all your tables and enumeration domains.

    Now you should be able to convert the sym column of your table to a numpy array of strings

    >>> np.array(q.t.sym)
    array(['A', 'B', 'C'], dtype=object)
    

    or to a numpy array of integers:

    >>> np.array(q.t.sym.data)
    array([0, 1, 2])
    

    You can also convert the entire table to a numpy record array in one go, but you will have to "map" it into the memory first:

    >>> np.array(q.t.select())
    array([('A', 1), ('B', 2), ('C', 3)], dtype=[('sym', 'O'), ('a', '<i8')])