Search code examples
amazon-web-servicesamazon-s3amazon-aurora

SELECT INTO OUTFILE S3 producing partial output

My requirement is to create text files out of database tables into S3 within a stored procedure. The size of the result set varies between 112KB to 100MB. I'm using SELECT INTO OUTFILE S3 command for the purpose. The table I'm trying to out into S3 has 10k rows amounting to a total of 4.5MB file size. When running the command, a single part file is created in S3 with only 200 rows. How can I fix this problem to get all 10k rows in s3 ?

Aurora MYSQL version = 2.11.2

The SQL query used -

SELECT * FROM my_database_table
INTO OUTFILE S3 's3://my-bucket/folder/mytable'
    FORMAT CSV HEADER
    MANIFEST ON
    OVERWRITE ON;
Solution

  • Figured out the mistake, Dataset Fetch Size was set as 200 as in the client tool. Once it was modified to a bigger number, the file got written as expected.