How to load multiple huge csv (with different columns) into AWS S3

Question

I have around 50 csv files each of different structure. Each csv file has close to 1000 columns. I am using DictReader to merge csv files locally, but it is taking too much time to merge. The approach was to merge 1.csv and 2.csv to create 12.csv. Then merge 12.csv with 3.csv. This is not the right approach.

for filename in inputs:
    with open(filename, "r", newline="") as f_in:
      reader = csv.DictReader(f_in)  # Uses the field names in this file

Since I have to finally upload this huge single csv to AWS, I was thinking about a better AWS based solution. Any suggestions on how I can import these multiple different structure csv and merge it in AWS?

Answer 1

Launch an EMR cluster and merge the files with Apache Spark. This gives you complete control over the schema. This answer might help for example.

Alternatively, you can also try your luck and see how AWS Glue handles the multiple schemas when you create a crawler.

You should copy your data to s3 in both cases.