Search code examples
javaamazon-web-servicesamazon-s3datadog

How to reduce response time from Amazon S3?

I have java method that gets last saved file from S3 bucket by getting list of all objects and sorting them by name (all files has timestamp in their name). The problem is that response time could be 13 seconds and up to 20, depends on how many objects in the bucket enter image description here

I guess problem lies in call rout - client - cloudflare - loadbalancer - ingress controler - my service container - ingress controler - loadbalancer - cloudflare - client, because when I call this method from localhost, response time is 1.5 ms in avarage

Is there a way to reduce response time?

Service method

public URL getLastSavedFile(Integer vehicleId) {
    try {
        S3ObjectSummary s3ObjectSummary = repository.downloadLastFile(String.valueOf(vehicleId));
        GeneratePresignedUrlRequest request = getPresignedUrl(s3ObjectSummary);
        log.info("Returned URL of last file from S3 storage, for vehicle - {}", vehicleId);
        return s3Client.generatePresignedUrl(request);
    } catch (IndexOutOfBoundsException e) {
        log.info("Failed to get last snapshot, the following - {} doesn't exist in storage", vehicleId);
        throw new RuntimeException(e);
    }
}

Repository

public S3ObjectSummary downloadLastFile(String vehicleId) {
    ObjectListing listing = s3client.listObjects( env.getProperty("INIT_S3_BUCKET"), vehicleId );
    List<S3ObjectSummary> objects = listing.getObjectSummaries();
    while (listing.isTruncated()) {
        listing = s3client.listNextBatchOfObjects (listing);
        objects.addAll (listing.getObjectSummaries());
    }
    objects.sort((o1, o2) -> o2.getKey().compareTo(o1.getKey()));
    return objects.get(0); //sorted list with most recent timestamp in the beginning
}

UPD:

FIXED! It appers that all problems were due to the maven dependencies, I had old version 1.12.382 of aws-java-sdk and when I change it to version 1.12.415 of aws-java-sdk-s3 - it fixed the problem and now response time is 1-2 seconds

Solution

  • FIXED! It appers that all problems were due to the maven dependencies, I had old version 1.12.382 of aws-java-sdk and when I change it to version 1.12.415 of aws-java-sdk-s3 - it fixed the problem and now response time is 1-2 seconds