Search code examples
pythonredisparquetpyarrow

Storing parquet file in redis

I want to know how I would store a parquet file as it is binary data in redis via python?

Background is, that I want check the fastest way of serving a parquet file of small size over the network. I believe object storage like s3 or any file system is slower. So it comes down to the fastest way of serving binary data over the network, but still have some database like structure as storage layer. Probably the data needs to live in the memory on the server side to send it over network directly into the clients side memory. (remember it is about small size files compressed with parquet around 30-50 Mb, so to big for simple file system reads and to small for big data technology, hdfs etc. )

Solution

  • Okay found it myself:

    1. Use python Pandas to write as bytes into memory

      bytes_data = df.to_parquet()

    2. Now having the compressed parquet format as bytes in memory one can send it to redis

      set("key", bytes_data)