Download large files using InputStream

Question

We have a Weblogic server running on Linux with multiple managed servers. The managed server I am concerned with has a max heap size of 1024MB. This server has multiple applications deployed on it. One of those applications processes response from a REST api to download files as big as 250MB. For files bigger than 50 MB, sometimes there are no issues but sometimes the server crashes with OOM error. Following is my code:

Client client = Client.create();          
WebResource webResource = client.resource(url.toString());
ClientResponse response = webResource.accept("application/json").get(ClientResponse.class);
if (response.getStatus() != 200) {
            throw new RuntimeException("Failed : HTTP error code : "+ response.getStatus());
        }
String output = response.getEntity(String.class);
byte[] fileContent = Base64.decodeBase64(output.getBytes());
if (fileContent != null) {
        OutputStream out = null;
        try {
            res.reset();
            out = res.getOutputStream();
            res.setContentType(contentType);
            res.setHeader("Content-Disposition", "inline; filename=" + fileName + "; size=" + String.valueOf(fileContent.length));
            res.setContentLength(fileContent.length);
            out.write(fileContent);
        } catch (Exception ex) {
            e.printStackTrace();
        } finally {
            out.flush();
            out.close();
        }
    } 

Since there were memory issues, I tried to take the InputStream approach. Following is the changed code:

Client client = Client.create();          
WebResource webResource = client.resource(url.toString());
ClientResponse response = webResource.accept("application/json").get(ClientResponse.class);
if (response.getStatus() != 200) {
        throw new RuntimeException("Failed : HTTP error code : "+ response.getStatus());
    }
InputStream source = response.getEntityInputStream(); // IS THIS OKAY?

if (source!= null) {
        OutputStream out = null;
        int count = 0;
        try {
            byte[] buffer = new byte[1024];
            int read = 0;
            res.reset();
            res.setContentType(contentType);
            res.setHeader("Content-Disposition", "inline; filename=" + fileName);
            out = res.getOutputStream();
            while ((read = source.read(buffer)) != -1) {
                out.write(buffer, 0, read);
                count++;
            }                
            System.out.println("COUNT: " + count);// For a 60MB file, this prints 86000. why?
        } catch (Exception ex) {
            e.printStackTrace();
        }finally{
            out.flush();
            out.close();
            source.close();
        }

This code doesn't run into OOM but the file fails to load/is corrupted. Is the problem related to the fact that it is a Base64 encoded response and I am not handling it properly? If yes, what should I do? The application is running on Java 7. Is my approach using InputStream to tackle OOM okay? The service isn't sending the response in chunks. What other ways could I go ahead? I am worried that I am running into errors (not always) just for 50MB of file. The server is a remote one and is handled by another group. How can I check if there are other reasons responsible for the server crashes?

Answer 1

I used Base64InputStream to decode the InputStream response and it worked fine.

InputStream stream = response.getEntityInputStream();
Base64InputStream bis = new Base64InputStream(stream);

Then, used bis to write the file. This approach now prints the count variable around 65000.