Search code examples
kdb

How to merge a list of tables with different columns in kdb/q into one table effeciently


I know I can do

raze table_list

But that require every single table in the table list with the same columns.

For my use case, I can accept adding columns with null values, if that column doesn't exist in all tables. In that case, I can do

(uj) over table_list

But this line of code is very slow. nowhere as fast as raze.

Is there an efficient way of merging all tables, even if they don't all have the same columns?


Solution

  • To counter my comment above (and to repeat what I wrote on the google group):

    You'll never get as fast as raze with conforming tables but using global append (if you can accept the creation of a global) can be an improvement on uj over:

    /conforming tabs
    n:1000000;
    tabs:{flip`col1`col2`col3`col4`col5`col6!(n?100i;n?100j;n?.z.D;n?10f;n?.z.P;n?5h)}each til 10
    
    q)\ts raze tabs
    82 570426736
    
    /non-conforming tabs
    nctabs:{flip(6?`4)!(n?100i;n?100j;n?.z.D;n?10f;n?.z.P;n?5h)}each til 10
    
    q)\ts a:(uj/)nctabs
    5560 11729374832
    
    /global append
    q)\ts {t::(uj/)0#'x;{t,:x}each x}nctabs
    3146 356520224
    
    q)a~t
    1b