Search code examples
javaspring-integration

SFTP Integration - SftpInboundFileSynchronizer - How not to download same file again

In my current application, using spring-batch job, I trigger the process to SFTP remote files to local directory, process it and delete files post processing.


    @Bean("ftpMessageSource")
    @EndpointId("streamInboundAdapter")
    @InboundChannelAdapter(channel = "sftpChannel", poller = @Poller(fixedDelay = "5000"), autoStartup = "false")
    public MessageSource<File> sftpMessageSource() {
        SftpInboundFileSynchronizingMessageSource source =
                new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
        source.setLocalDirectory(new File("sftp-inbound"));
        source.setAutoCreateLocalDirectory(true);
        source.setLocalFilter(new FileSystemPersistentAcceptOnceFileListFilter(metadataStore(),""));
        source.setMaxFetchSize(10);
        return source;
    }


    @Bean(name="fileStore")
    public PropertiesPersistingMetadataStore metadataStore() {
        PropertiesPersistingMetadataStore metadataStore = new PropertiesPersistingMetadataStore();
        metadataStore.setBaseDirectory("filestore");
        metadataStore.setFileName("filestore.properties");
        metadataStore.afterPropertiesSet();
        return metadataStore;
    }

As every file is processed, I make an entry of filename to fileStore.properties file.

    
metadataStore.put(file.getName(),file.getName());

One issue, I ran into next time processing (w/o restarting server and kicking off the same process again) is - process gets same set of files again for processing.

I do not want to SFTP processed files, can you please point out which configuration I am missing to avoid downloading same files again.

Solution

  • and delete files post processing.

    So, file does not exist in local directory any more. Since you don't filter remote files, they are downloaded again as a new local copy.

    The FileSystemPersistentAcceptOnceFileListFilter logic is based on the file.lastModified() and if it is different from an existing entry, it is replaced and therefore pushed downstream.

    Consider using an SftpPersistentAcceptOnceFileListFilter on the sftpInboundFileSynchronizer. This way the same file (if the same lastModified) is not going to be pulled from SFTP.

    See more in docs: https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/sftp.html#sftp-inbound