Is there a way to get an s3 object stream chunk by chunk using AWSSDK in dotnet?

Question

We have a system that gets an event from AWS S3 when a new object is PUT.

We have a bunch of services registered that we pass the GetObjectResponse to. Each service examines the GetObjectResponse via the ResponseStream. If one object "passes" on handling the object, it goes to the next one and so on. But, once the ResponseStream is started to be read by one service, the Position cannot be set to 0.

So, the GetObjectResponse is refreshed for each service to examine it. Then, once the service is found, it has to be refreshed again.

Is there any way to prevent these multiple calls to GetObjectAsync?

We get the object using:

var s3 = this.GetProvider().GetRequiredService<IAmazonS3>();
using GetObjectResponse getObjectResponse = await s3.GetObjectAsync(new GetObjectRequest() { BucketName = bucketName, Key = objectKey });

Answer 1

S3 is offering a feature called byte-range fetches, it allows us to fetch large files from s3 in chunk by chunk methods.

var s3 = this.GetProvider().GetRequiredService<IAmazonS3>();
using GetObjectResponse getObjectResponse = await s3.GetObjectAsync(new GetObjectRequest() { BucketName = bucketName, Key = objectKey, Range='bytes=START_BYTE_ID-END_BYTE_ID' }); #Place your values in