Search code examples
amazon-s3swisscomdev

Excessive usage of S3 ends with socket close?


This question is about the Swisscom Application Cloud specifically and not about Amazon.

My application uses 50 threads. In sum they make perhaps 25-200 requests per sec to S3. After running them for 10-30 seconds I start to get exceptions like this:

2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT com.amazonaws.AmazonClientException: Unable to execute HTTP request: Socket is closed
2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:956)
2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4041)
2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1581)
2016-10-29 14:36:58 [APP/PROC/WEB/0] OUT at <my_code_from_here>.putFile(S3Service.java:49)

After a restarting the app or waiting some minutes the issue is solved but as soon as I start and put load again on S3 I got these exceptions again after 10-30 seconds.

Is there any limit on the request rate?


Solution

  • There is no outbound traffic limitation or DoS protection.

    Swisscom AppCloud has a DoS policy for S3 (branded as Dynstrg, Vendor is EMC Atmos) access activated which intercepts requests after a certain level. This detection criteria is currently triggered by 200 TPS (Transactions per Seconds, TCP sessions) per source IP, then this IP is blocked for at least 120 seconds.

    Swisscom is currently discussing to increase those triggers.