Remove word after line from txt file is read

Question

I have this code which is used to read lines from a file and insert it into Postgre:

try {
            BufferedReader reader;
            try {
                reader = new BufferedReader(new FileReader(
                        "C:\\in_progress\\test.txt"));
                String line = reader.readLine();
                while (line != null) {
                    System.out.println(line);

                    Thread.sleep(100);
                    Optional<ProcessedWords> isFound = processedWordsService.findByKeyword(line);

                    if(!isFound.isPresent()){
                        ProcessedWords obj = ProcessedWords.builder()
                                .keyword(line)
                                .createdAt(LocalDateTime.now())
                                .build();
                        processedWordsService.save(obj);
                    }

                    // read next line
                    line = reader.readLine();
                }
                reader.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        catch (Exception e) {
            e.printStackTrace();
        }

How I can remove a line from the file after the line is inserted into SQL database?

Answer 1

The issues with the current code:

• Adhere to the Single responsibility principle. Your code is doing too many things: reads from a file, performs findByKeyword() call, prepares the data and hands it out to store in the database. It's hardly can be thoroughly tested, and it's very difficult to maintain.
• Always use try-with-recourses to get your recourses closed at any circumstances.
• Don't catch the general Exception type - your code should only catch thous exceptions, which are more or less expected and for which there's a clear scenario on how to handle them. But don't catch all the exceptions.

How I can remove a line from the file after the line is inserted into SQL database?

It is not possible to remove a line from a file in the literal sense. You can override the contents of the file or replace it with another file.

My advice would be to file data in memory, process it, and then write the lines which should be retained into the same file (i.e. override the file contents).

You can argue that the file is huge and dumping it into memory would result in an OutOfMemoryError. And you want to read a line from a file, process it somehow, then store the processed data into the database and then write the line into a file... So that everything is done line by line, all actions in one go for a single line, and as a consequence all the code is crammed in one method. I hope that's not the case because otherwise it's a clear XY-problem.

Firstly, File System isn't a reliable mean of storing data, and it's not very fast. If the file is massive, then reading and writing it will a take a considerable amount of time, and it's done just it in order to use a tinny bit of information then this approach is wrong - this information should be stored and structured differently (i.e. consider placing into a DB) so that it would be possible to retrieve the required data, and there would be no problem with removing entries that are no longer needed.

But if the file is lean, and it doesn't contain critical data. Then it's totally fine, I will proceed assuming that it's the case.

The overall approach is to generate a map Map<String, Optional<ProcessedWords>> based on the file contents, process the non-empty optionals and prepare a list of lines to override the previous file content.

The code below is based on the NIO2 file system API.

public void readProcessAndRemove(ProcessedWordsService service, Path path) {
    
    Map<String, Optional<ProcessedWords>> result;
    
    try (var lines = Files.lines(path)) {
        result = processLines(service, lines);
    } catch (IOException e) {
        result = Collections.emptyMap();
        logger.log();
        e.printStackTrace();
    }
    
    List<String> linesToRetain = prepareAndSave(service, result);
    writeToFile(linesToRetain, path);
}

Processing the stream of lines from a file returned Files.lines():

private static Map<String, Optional<ProcessedWords>> processLines(ProcessedWordsService service,
                                                                  Stream<String> lines) {
    return lines.collect(Collectors.toMap(
        Function.identity(),
        service::findByKeyword
    ));
}

Saving the words for which findByKeyword() returned an empty optional:

private static List<String> prepareAndSave(ProcessedWordsService service,
                                           Map<String, Optional<ProcessedWords>> wordByLine) {
    wordByLine.forEach((k, v) -> {
        if (v.isEmpty()) saveWord(service, k);
    });
    
    return getLinesToRetain(wordByLine);
}

private static void saveWord(ProcessedWordsService service, String line) {
    
    ProcessedWords obj = ProcessedWords.builder()
        .keyword(line)
        .createdAt(LocalDateTime.now())
        .build();
    service.save(obj);
}

Generating a list of lines to retain:

private static List<String> getLinesToRetain(Map<String, Optional<ProcessedWords>> wordByLine) {
    
    return wordByLine.entrySet().stream()
        .filter(entry -> entry.getValue().isPresent())
        .map(Map.Entry::getKey)
        .collect(Collectors.toList());
}

Overriding the file contents using Files.write(). Note: since varargs OpenOption isn't provided with any arguments, this call would be treated as if the CREATE, TRUNCATE_EXISTING, and WRITE options are present.

private static void writeToFile(List<String> lines, Path path) {
    try {
        Files.write(path, lines);
    } catch (IOException e) {
        logger.log();
        e.printStackTrace();
    }
}