Search code examples
javafiletextlines

Reading group of lines in HUGE files

I have no idea how to do the following: I want to process a really huge textfile (almost 5 gigabytes). Since I cannot copy the file into temporarily memory, I thought of reading the first 500 lines (or as many as fit into the memory, I am not sure about that yet), do something with them, then go on to the next 500 until I am done with the whole file.

Could you post an example of the "loop" or command that you need for that? Because all the ways I tried resulted in starting from the beginning again but I want to go on after finishing the previous 500 lines.

Help appreciated.

Solution 
  • BufferedReader br = new BufferedReader(new FileReader(file));
String line = null;
ArrayList<String> allLines = new ArrayList<String>();

while((line = br.readLine()) != null) {
     allLines.add(line);
     if (allLines.size() > 500) {
          processLines(allLines);
          allLines.clear();
     }
}

processLines(allLines);