I have a Cloud Storage bucket that has an event-driven Storage Transfer from S3. The files are in parquet format. I want to load all files (previous and incoming) to BigQuery. How should I do this?
As Mentioned by the @guillaume blaquiere solution for this is BigLake. For more details refer to this documentation. And also you can load all the parquet files from cloud storage to BigQuery. Check in this link to load parquet files.
Posting the answer as community wiki for the benefit of the community that might encounter this use case in the future.
Feel free to edit this answer for additional information