Search code examples
sqlsnowflake-cloud-data-platformparquet

How to dynamically create table in Snowflake getting schema from parquet file which stored in AWS

Could you help me to load a couple of parquet files to Snowflake.

I've got about 250 parquet-files which stored in AWS stage.

250 files = 250 different tables.

I'd like to dynamically load them into Snowflake tables.

So, I need:

  1. Get schema from parquet file... I've read that I could get the schema from parquet file using parquet-tools (Apache).
  2. Create table using schema from the parquet file
  3. Load data from parquet-file to this table.

Could anyone help me how to do that? Does exist the most efficient way to realize it? (by using GUI Snowflake, for example). Can't find it.

Thanks.

Solution

  • If the schema of the files is same you can put them in a single stage and use the Infer-Schema function. This will give you the schema of the parquet files. https://docs.snowflake.com/en/sql-reference/functions/infer_schema.html

    In case all files have different schema then I'm afraid you have to infer the schema on each file.