Search code examples
rustrust-polars

Is it possible to load Parquet data directly from memory?

I have a use case where I will be downloading Parquet data directly into memory (not into the filesystem). Is it possible to load these files as (lazy) dataframes from a Vec<u8>? instead of passing in the path?

Solution

  • Yes, you can:

    use polars::prelude::*;
use std::io::Cursor;

fn main() -> Result<()> {
    let mut df = df![
        "a" => [1, 2, 3]
    ]?;

    // write to buffer
    let mut buf = vec![];
    ParquetWriter::new(&mut buf).finish(&mut df)?;

    // read from buffer
    let reader = Cursor::new(&buf);
    let result = ParquetReader::new(reader).finish()?;

    // validate result
    df.frame_equal(&result);
    Ok(())
}