Search code examples
glassfish-3fileinputstreamfileoutputstreamjava-6

OutOfMemoryError while break zip file into two parts


My application breaks .zip file into two parts. It works fine when file size is under 100MB. If file size is more than 100MB it gives

java.lang.OutOfMemoryError
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:198)

I'm using glassfishV3 and JDK6

-Xms512m -Xmx512m is not enough

then I have set 
<jvm-options>-Xms1024m</jvm-options>
<jvm-options>-Xmx1024m</jvm-options>

but then it gives

"Error occurred during initialization of VM Could not reserve enough space for object heap Could not create the Java virtual machine."

public void create() {
    byte[] bFile = takeByteArr("file1.zip");
    byte[] bPart1 = Arrays.copyOfRange(bFile, 0, 100000);
    byte[] bPart2 = Arrays.copyOfRange(bFile, 100000, bFile.length);
    createFile(bPart1, "part1.zip");
    createFile(bPart2, "part2.zip");
}

void createFile(byte[] bFile, String path) {
    try {
        FileOutputStream fos = new FileOutputStream(path);
        fos.write(bFile);
        fos.close();
    } catch (FileNotFoundException ex) {
        System.out.println("FileNotFoundException : " + ex);
    } catch (IOException ioe) {
        System.out.println("IOException : " + ioe);
    }
}

byte[] takeByteArr(String filePath) {
    File file = new File(filePath);
    byte[] bFile = new byte[(int) file.length()];
    try {
        FileInputStream  fileInputStream = new FileInputStream(file);
        fileInputStream.read(bFile);
        fileInputStream.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
    return bFile;
}

Solution

  • As already suggested in a comment large files are not read in one part but into a smaller buffer. The buffer is then processed, in your case this would be saving the part file to disk, and reused in a loop which reads the file chunk by chunk. This method has the advantage that you only need as much memory as the size of your buffer is.

    The FileInputStream class has a method named read(byte[] b, int off, int len) which reads len number of bytes starting at offset off in byte array b from the InputStream. The method returns an int representing the number of bytes read during the method call and -1 if nothing could be read.

    Example:

        byte[] b = new byte[1024];
        int read = inputStream.read(b, 0, 1024);
    

    Now you have a byte array b which is 1024 bytes large and contains read bytes of content.

    You can loop through your file, but you have to pay attention to the last part. The byte array buffer will also have the size 100000 but I guess you don't want to create the last part with that size. Therefore you have to recreate the buffer with number of bytes which are really the last part of the content.

    Your code could look like this:

    public void create() throws IOException {
        FileInputStream in = new FileInputStream(new File("file1.zip"));
    
        byte[] b = new byte[100000];
        int read;
        int counter = 0;
        while ((read = in.read(b, 0, 100000)) != -1) {
            if(read != 100000) { // <- the last part
                b = Arrays.copyOfRange(b, 0, read);
            }
            createFile(b, "part" + counter++ + ".zip");
            b = new byte[100000];
        }  
    }