Lot of TIME_WAIT connections while using RestTemplate?

Question

I am using Spring RestTemplate to make a HTTP Calls to my RestService. I am using spring framework 3.2.8 version of RestTemplate. I cannot upgrade this since in our company we have a parent POM in which we are using Spring Framework version 3.2.8 so I need to stick to that.

Let's say I have two machines:

• machineA: This machine is running my code which uses RestTemplate as my HttpClient and from this machine I make HTTP Calls to my RestService which is running on a different machine (machineB). I have wrapped the below code around multithreaded application so that I can do load and performance testing on my client code.
• machineB: On this machine, I am running my RestService.

Now the problem I am seeing is whenever I run a load and performance testing on machineA - Meaning, my client code will make lot of HTTPClient calls to the RestService running on machineB very fast since the client code is getting called in a multithreaded way.

I always see lot of TIME_WAIT connections on machineA as shown below:

   298 ESTABLISHED
    14 LISTEN
     2 SYN_SENT
 10230 TIME_WAIT

  291 ESTABLISHED
   14 LISTEN
    1 SYN_SENT
17767 TIME_WAIT

    285 ESTABLISHED
   14 LISTEN
    1 SYN_SENT
24055 TIME_WAIT

I don't think it's a good sign that we have lot of TIME_WAIT connections here. Problem Statement:-

• What does this high TIME_WAIT connection mean here in a simple language on machineA?
• Is there any reason why this is happening with RestTemplate or is it just the way I am using RestTemplate? If I am doing anything wrong in the way I am using RestTemplate, then what's the right way to use it?

Do I need to set any keep-alive header or Connection:Close thing while using RestTemplate? Any inputs/suggestions are greatly appreciated as I am confuse what's going on here.

Below is how I am using RestTemplate in my code base in a simple way (just to explain the whole idea of how I am using RestTemplate):

public class DataClient implements Client {

    private final RestTemplate restTemplate = new RestTemplate();
    private ExecutorService executor = Executors.newFixedThreadPool(10);

    // for synchronous call
    @Override
    public String getSyncData(DataKey key) {        
        String response = null;
        Future<String> handler = null;
        try {
            handler = getAsyncData(key);
            response = handler.get(100, TimeUnit.MILLISECONDS); // we have a 100 milliseconds timeout value set
        } catch (TimeoutException ex) {
            // log an exception
            handler.cancel(true);
        } catch (Exception ex) {
            // log an exception
        }

        return response;
    }

    // for asynchronous call
    @Override
    public Future<String> getAsyncData(DataKey key) {
        Future<String> future = null;

        try {
            Task task = new Task(key, restTemplate);
            future = executor.submit(task); 
        } catch (Exception ex) {
            // log an exception
        }

        return future;
    }
}

And below is my simple Task class

class Task implements Callable<String> {

    private final RestTemplate restTemplate;
    private final DataKey key;

    public Task(DataKey key, RestTemplate restTemplate) {
        this.key = key;
        this.restTemplate = restTemplate;
    }

    public String call() throws Exception {
        ResponseEntity<String> response = null;

        String url = "some_url_created_by_using_key";

        // handling all try catch here
        response = restTemplate.exchange(url, HttpMethod.GET, null, String.class);

        return response.getBody();
    }
}

Answer 1

"TIME_WAIT" is the state that a TCP connection mantains during a configurable amount of time after closed (FIN/FIN reception). In this way, a possible "delayed" packet of one connection can not be mixed with a latter connection that reuses same port.

In a high-traffic test, it is normal to have a lot of them, but they should disappear after a few minutes test finished.