I need to write(append) huge string to flat file using java nio. The encoding is ISO-8859-1.
Currently we are writing as shown below. Is there any better way to do the same ?
public void writeToFile(Long limit) throws IOException{
String fileName = "/xyz/test.txt";
File file = new File(fileName);
FileOutputStream fileOutputStream = new FileOutputStream(file, true);
FileChannel fileChannel = fileOutputStream.getChannel();
ByteBuffer byteBuffer = null;
String messageToWrite = null;
for(int i=1; i<limit; i++){
//messageToWrite = get String Data From database
byteBuffer = ByteBuffer.wrap(messageToWrite.getBytes(Charset.forName("ISO-8859-1")));
fileChannel.write(byteBuffer);
}
fileChannel.close();
}
EDIT: Tried both options. Following are the results.
@Test
public void testWritingStringToFile() {
DiagnosticLogControlManagerImpl diagnosticLogControlManagerImpl = new DiagnosticLogControlManagerImpl();
try {
File file = diagnosticLogControlManagerImpl.createFile();
long startTime = System.currentTimeMillis();
writeToFileNIOWay(file);
//writeToFileIOWay(file);
long endTime = System.currentTimeMillis();
System.out.println("Total Time is " + (endTime - startTime));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
/**
*
* @param limit
* Long
* @throws IOException
* IOException
*/
public void writeToFileNIOWay(File file) throws IOException {
FileOutputStream fileOutputStream = new FileOutputStream(file, true);
FileChannel fileChannel = fileOutputStream.getChannel();
ByteBuffer byteBuffer = null;
String messageToWrite = null;
for (int i = 1; i < 1000000; i++) {
messageToWrite = "This is a test üüüüüüööööö";
byteBuffer = ByteBuffer.wrap(messageToWrite.getBytes(Charset
.forName("ISO-8859-1")));
fileChannel.write(byteBuffer);
}
}
/**
*
* @param limit
* Long
* @throws IOException
* IOException
*/
public void writeToFileIOWay(File file) throws IOException {
FileOutputStream fileOutputStream = new FileOutputStream(file, true);
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(
fileOutputStream, 128 * 100);
String messageToWrite = null;
for (int i = 1; i < 1000000; i++) {
messageToWrite = "This is a test üüüüüüööööö";
bufferedOutputStream.write(messageToWrite.getBytes(Charset
.forName("ISO-8859-1")));
}
bufferedOutputStream.flush();
fileOutputStream.close();
}
private File createFile() throws IOException {
File file = new File(FILE_PATH + "test_sixth_one.txt");
file.createNewFile();
return file;
}
Using ByteBuffer and Channel: took 4402 ms
Using buffered Writer : Took 563 ms
I don't think you will be able to get a strict answer without benchmarking your software. NIO may speed up the application significantly under the right conditions, but it may also make things slower. Here are some points:
rewind
and flip
? Seems like you are creating a new buffer for every string and just writing it to the channel. (If you go the NIO way, benchmark strategies that reuse the buffers instead of wrapping / discarding, I think they will do better).wrap
and allocateDirect may produce quite different buffers. Benchmark both to grasp the trade-offs. With direct allocation, be sure to reuse the same buffer in order to achieve the best performance.byte[]
or char[]
buffer with a reasonable size as well). I've seen many, many, many people discovering that NIO is no silver bullet.If you fancy some bleeding edge... Back to IO Trails for some NIO2 :D.
And here is a interesting benchmark about file copying using different strategies. I know it is a different problem, but I think most of the facts and author conclusions also apply to your problem.
Cheers,
Since @EJP tiped me that direct buffers wouldn't be efficient for this problem, I benchmark it myself and ended up with a nice NIO solution using nemory-mapped files. In my Macbook running OS X Lion this beats BufferedOutputStream
by a solid margin. but keep in mind that this might be OS / Hardware / VM specific:
public void writeToFileNIOWay2(File file) throws IOException {
final int numberOfIterations = 1000000;
final String messageToWrite = "This is a test üüüüüüööööö";
final byte[] messageBytes = messageToWrite.
getBytes(Charset.forName("ISO-8859-1"));
final long appendSize = numberOfIterations * messageBytes.length;
final RandomAccessFile raf = new RandomAccessFile(file, "rw");
raf.seek(raf.length());
final FileChannel fc = raf.getChannel();
final MappedByteBuffer mbf = fc.map(FileChannel.MapMode.READ_WRITE, fc.
position(), appendSize);
fc.close();
for (int i = 1; i < numberOfIterations; i++) {
mbf.put(messageBytes);
}
}
I admit that I cheated a little by calculating the total size to append (around 26 MB) beforehand. This may not be possible for several real world scenarios. Still, you can always use a "big enough appending size for the operations and later truncate the file.
To anyone looking for a modern (as in, Java 11+) solution to the problem, I would follow @DodgyCodeException's advice and use java.nio.file.Files.writeString
:
String fileName = "/xyz/test.txt";
String messageToWrite = "My long string";
Files.writeString(Paths.get(fileName), messageToWrite, StandardCharsets.ISO_8859_1);