for (int i=0; i<100000; i++) {
// REST API request.
restTemplate.exchange(url, HttpMethod.GET, request, String.class);
}
I have a situation where I have to request a resource for 100k users and it takes 70 minutes to finish. I tried to clean up my code as much as possible and I was able to reduce it only by 4 minutes).
Since each request is independent of each other, I would love to send requests in parallel (may be in 10s, 100s, or even 1000s of chunks which every finishes quickly). I'm hoping that I can reduce the time to 10 minutes or something close. How do I calculate which chunk size would get the job done quickly?
I have found the following way but I can't tell if the program processes all the 20 at a time; or 5 at a time; or 10 at a time.
IntStream.range(0,20).parallel().forEach(i->{
... do something here
});
I appericiate your help. I am open to any suggestions or critics!!
UPDATE: I was able to use IntStream and the task finished in 28 minutes. But I am not sure this is the best I could go for.
I used the following code in Java 8 and it did the work. I was able to reduce the batch job to run from 28 minutes to 3:39 minutes.
IntStream.range(0, 100000).parallel().forEach(i->{
restTemplate.exchange(url, HttpMethod.GET, request, String.class);
}
});