Search code examples
amazon-web-servicesamazon-s3julia

Put object/file to S3 (parquet or CSV)


I'm struggling to put parquet file into S3 using AWS or AWSS3, if someone could provide working example (I can connect to S3 and get files). The only one which doesn't throw any error is below but puts only file body (file of 20B but when I try to open that file from S3 I'm getting "ERROR: ArgumentError: invalid parquet: final bytes are "quet", expect "PAR1"):

p = S3Path("s3://bucket/folder/my_local_file.parquet")  
write(p, "my_local_file.parquet")

Solution

  • It appears that you want to upload a file from the local disk to Amazon S3 using Julia.

    From Unable to Create Object with Content in AWS S3 Bucket : r/Julia:

    using AWS: @service
    @service S3
    
    function upload_file_to_s3(bucket_name, file_path)
        file_name = basename(file_path)
        open(file_path, "r") do file
            S3.put_object(bucket_name, file_name, Dict("Body" => read(file, String)))
        end
        println("File $file_name uploaded to $bucket_name.")
    end
    
    template_file="./README.md"
    bucket_name="my-unique-bucket-name"
    upload_file_to_s3(bucket_name, template_file)