Big performance degrade when upgrading an aws-lambda from java-11 to java-17

Question

We recently updated our aws-lambda java runtime from java-11 to java-17 (the update was just switch of the runtime environment so it still runs code compiled with java-11). This had lead to one of our lambdas that processes large files started to timeout.

The lambda reads around 1GB json file from S3, converts it to XML while doing schema validation and writes it back to S3 ending up at around 1.5GB, all this is done using io streams (so we don't keep the full file in memory).

I have tried to isolate the issue and have written a minimal function that just reads the source file from S3 and does nothing else and my testing show that java-17 takes at least twice as long to read for a warm run.

We are using aws-sdk v1 (1.12.512) and the lambda is plain with no snapstart, have tried different memory configurations. I have tried both x86_64 amd arm64. In all cases I java-11 runs approximately twice as fast as java-17.

The code test code I'm using is:

package lambda;

import com.amazonaws.regions.Regions;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.S3Object;

import java.io.IOException;
import java.io.InputStream;
import java.time.Duration;
import java.util.Map;

import static java.lang.System.getenv;

public class Handler implements RequestHandler<Map<String, String>, String> {

    static final AmazonS3 s3 = AmazonS3ClientBuilder.standard().withRegion(Regions.EU_WEST_1).build();

    @Override
    public String handleRequest(Map<String, String> input, Context context) {
        long start = System.currentTimeMillis();
        long bytesRead = 0;
        try (S3Object s3o = s3.getObject(getenv("BUCKET"), getenv("KEY"))) {
            InputStream is = s3o.getObjectContent();
            byte[] buffer = new byte[1024 * 1024];
            int read;
            while ((read = is.read(buffer)) != -1) {
                bytesRead += read;
            }
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        Duration d = Duration.ofMillis(System.currentTimeMillis() - start);
        String msg =  String.format("Read %d bytes from S3 in %s %d bytes/sec", bytesRead, d, bytesRead / d.getSeconds());
        context.getLogger().log(msg);
        return msg;
    }
}

Sample output from the logs (these are warm invokes):

Java-17:

START RequestId: 76d41820-40ef-4afb-8f8e-8c65de7eb6fc Version: $LATEST
Read 946586484 bytes from S3 in PT40.242S 23664662 bytes/sec
END RequestId: 76d41820-40ef-4afb-8f8e-8c65de7eb6fc
REPORT RequestId: 76d41820-40ef-4afb-8f8e-8c65de7eb6fc  Duration: 40245.11 ms   Billed Duration: 40246 ms   Memory Size: 2048 MB    Max Memory Used: 141 MB 

Java-11:

START RequestId: bb994f7a-5e31-4a1d-a656-7e8b2fa0c377 Version: $LATEST
Read 946586484 bytes from S3 in PT21.095S 45075546 bytes/sec
END RequestId: bb994f7a-5e31-4a1d-a656-7e8b2fa0c377
REPORT RequestId: bb994f7a-5e31-4a1d-a656-7e8b2fa0c377  Duration: 21096.62 ms   Billed Duration: 21097 ms   Memory Size: 2048 MB    Max Memory Used: 174 MB 

Is anyone else experiencing similar performance degrades on java-17 and any hints on how to fix/optimize this?

I did try to google this and mostly found blog post telling the performance on java-17 is greater than java-11.

Answer 1

The JVM configuration for tiered compilation was changed in the Java 17 managed runtime.

You could change it back to the previous configuration with a environment variable.

JAVA_TOOL_OPTIONS set to "-XX:-TieredCompilation"

You can read about the changes on the Java 17 AWS blog