FileSystemPersistentAcceptOnceFileListFilter Not Picking Up File

Question

I am using OpenJDK Java 17, Spring Integration with Spring Boot 2.7.4. Watching a directory for files with the code below, I could see that the metadata store table was updated with the file and its timestamp. But it never got to the fileChannel code for processing. Timing issue perhaps?? This app has been running a few months with no issues before today. I did a touch command on the file and then it got triggered.

Any suggestions? Thanks in advance for any assistance.

@Bean
    public MessageChannel fileChannel() { return new DirectChannel(); }

@Bean
    @InboundChannelAdapter(value = "fileChannel", poller = @Poller(fixedDelay = "30000"))
    public MessageSource<File> watchSourceDirectory() {
        FileReadingMessageSource source = new FileReadingMessageSource();
        source.setDirectory(new File(appConfig.getLocal().getSourceDir()));
        source.setAutoCreateDirectory(true);
        CompositeFileListFilter<File> compositeFileListFilter = new CompositeFileListFilter<>();
        compositeFileListFilter.addFilter(new RegexPatternFileListFilter(appConfig.getLocal().getFilePattern()));
        compositeFileListFilter.addFilter(new LastModifiedFileListFilter(10));
        compositeFileListFilter.addFilter(new FileSystemPersistentAcceptOnceFileListFilter(metadataStore, ""));
        source.setFilter(compositeFileListFilter);
        return source;
    } // end watchSourceDirectory()

Answer 1

There is a problem with CompositeFileListFilter: it does apply polled files to all the filters, but return only those which remain in the original list:

public List<F> filterFiles(F[] files) {
    Assert.notNull(files, "'files' should not be null");
    List<F> results = new ArrayList<>(Arrays.asList(files));
    for (FileListFilter<F> fileFilter : this.fileFilters) {
        List<F> currentResults = fileFilter.filterFiles(files);
        results.retainAll(currentResults);
    }
    return results;
}

Consider to use a ChainFileListFilter instead. Its logic is to call the next filter only with already filtered files:

public List<F> filterFiles(F[] files) {
    Assert.notNull(files, "'files' should not be null");
    List<F> leftOver = Arrays.asList(files);
    for (FileListFilter<F> fileFilter : this.fileFilters) {
        if (leftOver.isEmpty()) {
            break;
        }
        @SuppressWarnings("unchecked")
        F[] fileArray = leftOver.toArray((F[]) Array.newInstance(leftOver.get(0).getClass(), leftOver.size()));
        leftOver = fileFilter.filterFiles(fileArray);
    }
    return leftOver;
}

So, your FileSystemPersistentAcceptOnceFileListFilter is not going to be updated for nothing.

This is what is going on in your case:

1. You got RegexPatternFileListFilter and probably all the files are accepted. No state.
2. Your files are not accepted because they are not old enough. No state.
3. Your files are accepted because they are not present in the store. There is a state.
4. But since last modified filter has already remove your files from the list, the result of the accept once is ignored.

Earlier it probably worked because files were old enough or you didn't have that LastModifiedFileListFilter.