Search code examples
javaduplicatesfile-writing

Is there a way to avoid writing duplicate lines to a file in Java?


I have a simple question:

I have some code that writes lines of data to a text file in Java. One of the method calls that writes data to the file is located within a method that is called more frequently than the data itself changes.

As a result, the text file contains numerous duplicates of each line that is written from this method. I know I can remove these duplicates from the file after the fact, but I am wondering if anyone knows of a way to avoid writing duplicate lines to a file in the first place. Thanks for reading.

The method that writes data to the file:

@Override
public void writeMessage(String location, double time, String action) {
    write(location + " " + time + " " + action);
}

And the method that calls it:

public int receiveMessage(Message m, DTNHost from) {
    int retVal = this.router.receiveMessage(m, from);

    if (retVal == MessageRouter.RCV_OK) {
        m.addNodeOnPath(this);  // add this node on the messages path
    }

    writeMessage(untranslate().toString(), SimClock.getTime(), "receiving message");

    return retVal;
}

untranslate.toString() returns a pair of xy coordinates as a string, and SimClock.getTime() returns a timestamp.

My solution was to define a String called lastMsg, initialize it as null in the constructor of the class, and then to check if the ID of the message parameter in receiveMessage() has changed before writing to the file:

public int receiveMessage(Message m, DTNHost from) {
    String currentMsg = m.getId();
    int retVal = this.router.receiveMessage(m, from);

    if (retVal == MessageRouter.RCV_OK) {
        m.addNodeOnPath(this);  // add this node on the messages path
    }
    if (!(currentMsg.equals(this.lastMsg))) {
        writeMessage(untranslate().toString(), SimClock.getTime(), "receiving message");
        this.lastMsg = currentMsg;
    }
    return retVal;
}

Solution

  • Seems trivial; that method should check 'previous line written'. If the same, return;, doing nothing. If not, write it, and change the 'previous line written' field.

    If you're calling this method from multiple threads, you're already in trouble (if two threads both call a method that writes 'Hello!' to a file, you could end up with a file with a line: HeHellllo!o! - you need some synchronization. Which then guarantees the field is synced too. So:

    private String lastLineWritten = null;
    
    public /* synchronized - you may need this */ void writeLog(String line) {
        if (line == null) throw new NullPointerException();
        if (line.equals(lastLineWritten)) return;
        lastLineWritten = line;
        try (BufferedWriter bw = Files.newBufferedWriter(path, StandardCharsets.UTF_8, StandardOpenOption.APPEND)) {
            bw.write(line);
        }
    }