Apache Arrow table from iostream or memory buffer

Question

I have some code that retrieves a parquet file from AWS S3 using the AWS API. The result is a std iostream:

   std::basic_iostream< char, std::char_traits<char>> 

From this I want to create an Apache Arrow Table without saving the iostream to a local file and then loading. However, I don't see anything in the API to help. Arrow only provides interfaces for loading parquets from a file.

Answer 1

Preliminary support for direct S3 support has been added to master. You should be able to produce a file that can be passed in for Parquet reading using the new API. My understanding is this was also built on-top of the official S3 client, so you could also adapt the code for your use-case.