I’m working on a system where I need to frequently append small amounts of data (about 1 MB) to large files (2 GB or more). The files are stored in the cloud, and I need a solution that allows me to easily add data to existing files without needing to download and re-upload the entire file each time.
My Requirements: Frequent appending of small data chunks (1 MB). File sizes will be in the GB range (2 GB or larger). The solution must support low-latency file access for multiple servers. Ideally, I want to avoid downloading the entire file and uploading it again when appending data.
Amazon S3 is not suitable for your requirements because it is not possible to 'append' data to objects stored in S3. You would need to download, modify and upload the file -- however, you stated that you do not wish to do this.
Therefore, Amazon EFS would be a better option than Amazon S3 because EFS operates like a normal file system where you can edit existing files and it is accessible on multiple servers.