Search code examples
amazon-web-servicesrdsbigdata

Upload multiple local csv files to AWS RDS


What's the fastest way to upload multiple local csv files to a AWS RDS? I have 100 Gbs of data in thousands of csv files sit on a local machine.


Solution

  • To expand on @spg's answer: He is right in that EC2 to RDS will be MUCH faster than local to RDS.

    1. So launch an EC2 instance and install Postgres client psql.
    2. Use scp to copy the CSV file directly to the EC2 instance.
    3. Use the psql /copy command to import items from CSV:

      $psql target-db ^ -U <admin user> ^ -p <port> ^ -h <DB instance name> ^ -c "\copy source-table from '/path/to/source-table.csv' with DELIMITER ','"

    See AWS Documentation