Read a file(>150MB) and return the file content as ByteArrayOutputStream

Question

I am trying to read a large file (>150MB) and return the file content as a ByteArrayOutputStream. This is my code...

private ByteArrayOutputStream readfileContent(String url) throws IOException{

    log.info("Entering readfileContent ");
    ByteArrayOutputStream writer=null;
    FileInputStream reader=null;

    try{
        reader = new FileInputStream(url);
        writer = new ByteArrayOutputStream();

        byte[] buffer = new byte[1024];

        int bytesRead = reader.read(buffer);
        while (bytesRead =  > -1) { 
            writer.write(buffer, 0, bytesRead);
            buffer = new byte[1024];
        }

    }
    finally {
        writer.close();
    }

    log.info("Exiting readfileContent ");
    return writer;
}

I am getting an java.lang.OutOfMemoryError: Java heap space exception. I have tried increasing the java heap size, but it still happens. Could someone please assist with this problem.

Answer 1

Your approach is going to use at least the same ammount of memory as the file, but because ByteArrayOutputStream is using a byte array as storage, it'll potentially have to resize itself 150,000 times (150 meg/1024k buffer) which is not efficient. Upping the heap size to 2* your file size and increasing the size of buf to something much larger may allow it to run, but as other posters have said, it's far better to read form the file as you go, rather than read it in as a String.