how can i synchronize a directory with subdirectories over ftp using spring integration with spring batch?

Question

I've been trying to use SftpInboundFileSynchronizer with a remote directory that contains a subdir, say /myfiles/mysubdir/lefile.txt, I have set a filter to grab the files inside the dirs:

mysync.setRemoteDirectory("myfiles/");
mysync.setFilter(new SftpRegexPatternFileListFilter(".*\\.txt$"));

And then a SftpInboundFileSynchronizingMessageSource as my InboundChannelAdapter I have set on the SftpInboundFileSynchronizingMessageSource a RecursiveDirectoryScanner as scanner and i have no set limit to the depth or the amount of files to retrieve. I also set a FOLLOW_LINKS fileVisitOption on the scanner for good measure.

I am only able to pull files into the local directory from the myfiles path, but anything deeper is not copied to the local dir. I can't for the life of me figure out if there is something I'm not doing.

EDIT: What would the InboundChannelAdapter contain if I'm only going to send "/" as the directory to check with mget -R?

@Bean
@InboundChannelAdapter(value = "sftpChannel", poller = @Poller(fixedDelay = "10"))
public MessageSource<?> myMessageSource() {
}

@Bean(name = "myGateway")
@ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
    SftpOutboundGateway gateway =
        new SftpOutboundGateway(sftpSessionFactory(), "mget", "'myfiles/*'");

    gateway.setOutputChannelName("listSplitter");
    gateway.setOptions("-R");
    gateway.setAutoCreateLocalDirectory(true);

    myLocalPath = Paths.get(myLocalParentDir).toRealPath().toString();

    gateway.setLocalDirectory(new File(myLocalPath));

    SftpRegexPatternFileListFilter regexFilter = new regexFilter("^.*\\.txt");
    regexFilter.setAlwaysAcceptDirectories(true);
    regexFilter.setFilter(sftpRegexPatternFileListFilter);

    return gateway;
}

Answer 1

Recursion of the remote file system is not supported by the inbound synchronizer; use an SftpOutboundGateway (request/reply) instead, with a recursive mget command.

By default, files existing in the local directory are not re-fetched; you can control that with the FileExistsMode.