Reading text file to string without huge memory consumption

Question

I've tried to measure performance of several approaches to read a file into string using NIO (slowest for reading single file), BufferedInputStream and reading the file line after line (600 ms average per pass) and then this stream using Filereader and an array with fixed size acting as a buffer (fastest)

File was 95 MB of pure text in windows .txt file format. Converting chars to string really is the bottleneck, but what I noticed is HUGE memory consumption of this method. For 95 MB of lorem ipsum, this consumes up to 1 GB of RAM. I haven't found why.

What I have tried with no effect:

Issuing Garbage Collector by calling System.gc() Setting all the pointer variables to null before method ends (but they should be anyway, they are defined only within method).

private void testCharStream() {
            File f = f = new File("c:/Downloads/test.txt");
    long oldTime = System.currentTimeMillis();
    char[] cbuf = new char[8192];
    StringBuilder builder = new StringBuilder();
    try {

        FileReader reader = new FileReader(f);

        while (reader.read(cbuf) != -1) {
            builder.append(cbuf);
        }

        reader.close();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    long currentTime = System.currentTimeMillis();

    System.out.println(currentTime - oldTime);
}

Answer 1

Try Apache Commons IO: http://commons.apache.org/proper/commons-io/ I didn't benchmark it but I think the code is optimised.