Trying to consume HUGE JSON data using Spring RestTemplate resulting in java heap space exception

Question

I am using Resttemplate to consume a huge json from an external server. My code works fine when the dataset is not huge but as soon as I run it against the full dataset, I am unable to map the response to the bean class. Below is my code.

public void createCustomCsv(String name, String password, String serverUrl1, String serverUrl2, String propLocation)
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);

RestTemplate restTemplate = new RestTemplate();
restTemplate.setRequestFactory(requestFactory);

HttpHeaders httpHeaders = customHeaders.createCustomHeaders(name, password);

List<HttpMessageConverter<?>> messageConverters = new ArrayList<HttpMessageConverter<?>>();
 //Add the required converters
messageConverters.add(new MappingJackson2HttpMessageConverter());
messageConverters.add(new StringHttpMessageConverter());
 //Add the message converters to the restTemplate
restTemplate.setMessageConverters(messageConverters); 

ResponseEntity<MyDataBean> responseEntity1;
ResponseEntity<MyDataBean> responseEntity2;
try {
    long startTime = System.currentTimeMillis();
    jLog.debug("Start mapping to Pojo :: " +startTime);
    responseEntity1 = restTemplate.exchange(serverUrl1, HttpMethod.GET, httpEntity, MyDataBean.class);
 responseEntity2 = restTemplate.exchange(serverUrl2, HttpMethod.GET, httpEntity, MyDataBean.class);


MyDataBean sampleDataBeanServer1 = responseEntity1 .getBody();
MyDataBean sampleDataBeanServer2 = responseEntity2 .getBody();


processCustomData(sampleDataBeanServer1 , propLocation);
processCustomData(sampleDataBeanServer2 , propLocation);
} catch (RestClientException e) {

    jLog.debug("Something went wrong with ws call:::" +e);
}
}

I have tried for hours and searched in SO trying to figure out a way to parse the response data into the Pojo class without getting memory exceptions. I understand that this might be because the entire response is being retained in memory to map it to the Pojo. The below code is where the memory problem is found.

 responseEntity1 = restTemplate.exchange(serverUrl1, HttpMethod.GET, httpEntity, MyDataBean.class); 

Based on docs, I have also set the BufferRequestBody to false so that the response doesn't get loaded into memory. I am not sure why the behaviour is different as compared to the docs.

 SimpleClientHttpRequestFactory requestFactory = new  SimpleClientHttpRequestFactory();
 requestFactory.setBufferRequestBody(false);

It would be great if someone is patient and kind to help me out of this soup! I am open to any third party library suggestions as well.

P.N: The above code works perfect when dataset is not huge. Also omitted the bean class code for brevity.

Answer 1

To handle large files you would have to download the file by reading directly from the response stream. This link might help. And then read from the downloaded file in batches using buffered reader and have them in your entities and then process them.

You wouldn't be be able to process them in one batch as it wouldn't fit in memory.