Search code examples
javaperformanceweb-servicesrestbulk

Fast Rest Calls Execution

This question has two parts

Part 1

I need to make bulk Rest API Calls for Third Party API, i.e around 300K per hour, so What's the best way to optimize my code to get/manage response with in time. As in a test I tried to generate 1000 Threads and called below code to get response in minimum time, but the result was pretty disappointing. Some of the calls responded successfully but many of the calls returned timeout error.

        HttpClient client = new DefaultHttpClient();
        HttpGet httpGet = new HttpGet("http://www.example.com");

        HttpResponse response;
        response = client.execute(httpGet);
        System.out.println(response.getStatusLine().getStatusCode());
        HttpEntity entity = response.getEntity();
        String result = getStringFromInputStream(entity.getContent());

I know that timeout error can be due to server side but for instance server returns the response and all my 1000 threads respond within 5mins time, then still I wouldn't be able to achieve 300k calls per hour. Secondly can I optimize this code in some other way?

Part 2

I tried this with Java, I am just curious to know would it give better result if I use language other than java? i.e. javascript/python/PHP?

Update 1

My System Specs for the Tests conducted

  • Ubuntu 14.04 64 bit
  • 4Gb RAM
  • 500 GB Hard Drive
  • Intel® Core™ i5-2430M CPU @ 2.40GHz × 4
  • Internet connection = 1 MB :(

Update 2

My server specs to conduct tests in future/practical

  • Cent OS 6.5
  • 1 GB RAM
  • 1 CPU Core
  • 40 Gbit Network In
  • 125 Mbit Network Out :)

Please guide. Thank you.

Solution

  • Apache HTTPCommons 4.3 Fluent API did the trick for me and I was able to serve far more request than stated above.