Search code examples
scalaapache-sparkapache-spark-sqlsnappydata

Theta Sketch (Yahoo) on SnappyData


How to store Theta Sketch (Yahoo) on SnappyData's table instead of write to file? Because I generate billions of sketches every day and need to keep many millions of sketches online for real-time queries. Can anyone help me? Thanks.


Solution

  • Can’t you store these in a column with blob data type ? If writing out from a spark program and managing the sketches in a DF , I would think df.write.format(“column”).saveAsTable should work.

    Else, serialize (sketch.compact().toByteArray()) and store in blob column using sql.