Search code examples
ckdb+

KDB: Upserting only specified columns


https://code.kx.com/q/wp/capi/#publishing-to-a-kdb-tickerplant

Schema :

trade:([]
  time:`timespan$();
  sym:`symbol$();
  price:`float$();
  size:`long$()`
  )

Sample tickerplant publisher code :

    K singleRow= knk(3, ks((S) "ABC"), kf(10.0), kj(20));
    // Perform single row upsert, tickerplant will add timestamp column itself
    result= k(handle, ".u.upd", ks((S) "trade"), singleRow, (K)0); 

Now let's assume that this table is keyed by sym column. Now when i execute this function it calls upsert, but what I need is on top of upsert functionality (insert if the key doesn't exist and update if key exists), what I also need is during upsert I specify only certain columns only which means during upsert if the key exists it updates only that column that specified and others remain unchanged. Can i modify it in the C code ?

For example pseudocode :

    K singleRow= knk(3, ks((S) "ABC"), kj(20));
    // Perform single row upsert, tickerplant will add timestamp column itself
    result= k(handle, ".u.upd", ks((S) "trade"), singleRow, (K)0); 

As you can see i only specified size column value, so what I expect to do is that if the key exists (in this case ABC) then it only updates size column and since i didnt specified price column it should be the same as previous value of that.


Solution

  • You are sending a list to the kdb+ tickerplant, but there is no way for the tickerplant to know what column the values refer to if you don't send them all. For example, is

    (`ABC;2024.01.06D13:19:40.151165000;20)
    

    sym,time,price or sym,time,size? Sure you could type check (float for price, long for sizes in your example - but you might also need to handle fractional shares, which means size would be a float also).

    In my opinion, your best option is to always send all of the fields from the feedhandler, defaulting any missing entries, so that you always send a list with an entry for each column. You should also be aware that if you are keying your table in the tickerplant and you have batching enabled - intra batch updates may overwrite previous updates.

    If you can't do that for whatever reason, your next best option would be to modify the .u.upd function to accept table name, data and a list of column names you're sending. You can then combine the column names and data in the .u.upd function - upsert then handles the functionality you want. e.g:

    q)// sending all values
    q)trade upsert`sym xkey enlist`sym`time`price`size!(`ABC;.z.p;10.0;20)
    sym| time                          price size
    ---| ----------------------------------------
    ABC| 2024.01.06D13:27:43.368951000 10    20
    q)// sending a subset of values
    q)trade upsert`sym xkey enlist`sym`size!(`ABC;20)
    sym| time price size
    ---| ---------------
    ABC|            20
    

    If you need to carry over previous values when some are missing, you can use fills in your rdb: (your table needs to be unkeyed for this):

    q)trade:trade upsert enlist`sym`time`price`size!(`ABC;.z.p;10.0;20)
    q)trade:trade upsert enlist`sym`time`size!(`ABC;.z.p;20)
    q)trade
    sym time                          price size
    --------------------------------------------
    ABC 2024.01.06D13:31:34.836794000 10    20
    ABC 2024.01.06D13:31:48.141010000       20
    q)fills trade
    sym time                          price size
    --------------------------------------------
    ABC 2024.01.06D13:31:34.836794000 10    20
    ABC 2024.01.06D13:31:48.141010000 10    20