Faster Way to write files in Java

Question

I am using the current function to read a large file and then distribute it to different shorter files. It takes 13 mins for a 100 MB file.

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.Scanner;


public class DivideData {

public static void main(String[] args) throws IOException {
    Scanner data =  new Scanner(new File("D:\\P&G\\March Sample Data\\march.txt"));

    long startTime = System.currentTimeMillis();
    while(data.hasNextLine()){                          
        String line = data.nextLine();
        String[] split = line.split("\t");
        String filename = "D:\\P&G\\March Sample Data\\" + split[0] + " "+ split[1]+ ".txt";
        //System.out.println((filename));
        //System.out.println(line); 

        FileWriter fw = new FileWriter(filename,true); //the true will append the new data
        fw.write(line);//appends the string to the file
        fw.write('\n');
        fw.close();         

    }
    long stopTime = System.currentTimeMillis();
    System.out.println(stopTime - startTime);
    data.close();
    System.out.println("Data Scueessfully Divided!!");
}

}

I want to know what I can do to reduce the time it takes.

Answer 1

Move the FileWriter open and close outside the loop,

FileWriter fw = new FileWriter(filename,true); // <-- here!
while(data.hasNextLine()){                          
    String line = data.nextLine();
    String[] split = line.split("\t");
    String filename = "D:\\P&G\\March Sample Data\\" + split[0] + " "
            + split[1]+ ".txt";
    //System.out.println((filename));
    //System.out.println(line); 
    // FileWriter fw = new FileWriter(filename,true);

Otherwise it has to open the file and seek to the end for every line of input!

Edit

I noticed you don't have the filename until in your loop. Let's use a Map to keep a cache.

FileWriter fw = null;
Map<String, FileWriter> map = new HashMap<>();
while (data.hasNextLine()) {
    String line = data.nextLine();
    String[] split = line.split("\t");
    String filename = "D:\\P&G\\March Sample Data\\" + split[0] + " "
            + split[1] + ".txt";
    // System.out.println((filename));
    // System.out.println(line);
    if (map.containsKey(filename)) {
        fw = map.get(filename);
    } else {
        fw = new FileWriter(filename, true);
        map.put(filename, fw);
    }
    // ...
}
for (FileWriter file : map.values()) {
    file.close();
}