Search code examples
javacompressiongzipinputstreamgzipoutputstream

java GZIPInputStream and GZIPOutputStream not working as expected


I am trying to write a series of Long's to a GZIPOutputStream, hoping to decompress the numbers later.
The following program works fine when I try it with a small number of Long's, but it will throw an exception with many Long's, like (10240).

Please see and run this code:

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.nio.ByteBuffer;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

public class GzipTest {
    private static final String filename = "data.gz";

    public static void main(String... args) throws Exception {
        test(32);  // OK
        test(10240);  // Why won't this work? (unexpected read: 6)
    }

    public static void test(int max) throws Exception {
        System.out.println("testing with " + max);
        FileOutputStream fos = new FileOutputStream(filename);
        GZIPOutputStream gos = new GZIPOutputStream(fos, 4096);
        long num = 10241000L;
        for (int i = 0; i < max; ++i) {
            gos.write(long2bytes(num + i));
        }
        gos.close();
        fos.close();

        GZIPInputStream gis = new GZIPInputStream(new FileInputStream(filename), 4096);
        byte[] buf = new byte[Long.BYTES];
        for (int i = 0; i < max; ++i) {
            int nread = gis.read(buf);
            if (nread != Long.BYTES) {
                throw new RuntimeException("unexpected read: " + nread);
            }
            long value = bytes2long(buf);
            if (value != num + i) {
                throw new RuntimeException("unexpected value: " + value);
            }
        }
        gis.close();
    }

    static byte[] long2bytes(long num) {
        ByteBuffer buf = ByteBuffer.allocate(Long.BYTES);
        buf.putLong(num);
        return buf.array();
    }

    static long bytes2long(byte[] bytes) {
        return ByteBuffer.wrap(bytes).getLong();
    }
}

The outputs:

testing with 32
testing with 10240
Exception in thread "main" java.lang.RuntimeException: unexpected read: 6
    at GzipTest.test(GzipTest.java:31)
    at GzipTest.main(GzipTest.java:12)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

Solution

  • Th code works without any problems if you make the changes proposed by Peter Lawrey:

     try (DataInputStream in = new DataInputStream(new GZIPInputStream(new FileInputStream(filename), 4096))) {
            byte[] buf = new byte[Long.BYTES];
            for (int i = 0; i < max; ++i) {
                in.readFully(buf);
                long value = bytes2long(buf);
                if (value != num + i) {
                    throw new RuntimeException("unexpected value: " + value);
                }
            }
        }
    

    Or you can simplify everything to:

     try (DataInputStream in = new DataInputStream(new GZIPInputStream(new FileInputStream(filename), 4096))) {
            for (int i = 0; i < max; ++i) {
                long value = in.readLong();
                if (value != num + i) {
                    throw new RuntimeException("unexpected value: " + value);
                }
            }
        }