Spring-Integration: multithreaded creation of sftp directories with common root fails

Question

I have multiple threads which upload sftp files through an Sftp.outboundAdapter. Each thread needs to create a directory path with a common root. I get an exception which tells me that the common root exists:

org.springframework.messaging.MessageDeliveryException: Failed to transfer file
...
Caused by: org.springframework.core.NestedIOException: failed to create remote directory '/my/import/de3fb731-6a56-11e9-bfd2-0242ac140022'.; nested exception is 4: Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist. 
    at org.springframework.integration.sftp.session.SftpSession.mkdir(SftpSession.java:232)
    at org.springframework.integration.file.remote.RemoteFileUtils.makeDirectories(RemoteFileUtils.java:71)
    at org.springframework.integration.file.remote.RemoteFileTemplate.sendFileToRemoteDirectory(RemoteFileTemplate.java:545)
    at org.springframework.integration.file.remote.RemoteFileTemplate.doSend(RemoteFileTemplate.java:337)
    ... 150 common frames omitted
Caused by: com.jcraft.jsch.SftpException: Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist. 
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2873)
    at com.jcraft.jsch.ChannelSftp.mkdir(ChannelSftp.java:2182)
    at org.springframework.integration.sftp.session.SftpSession.mkdir(SftpSession.java:229)
    ... 153 common frames omitted

I guess there is a race condition in RemoteFileUtils#makeDirectories when it determines the pathsToCreate. Both threads determine that the path segment needs to be created, but when both try to create it, the second one fails.

A possible workaround is to create the common path in advance, but that is clumsy because there is no such command in the Sftp.outboundGateway, It seems I have to upload a dummy file with autoCreateDirectories and delete the dummy file afterwards.

Is there a better way to solve this? Would it make sense to add an mkdirs command to the sftp outbound gateway? Should there be a flag to ignore errors due to existing directories when autoCreateDirectories is true?

Answer 1

A possible workaround is to create the common path in advance, but that is clumsy because there is no such command in the Sftp.outboundGateway

You could simply call RemoteFileUtils.makeDirectories yourself during initialization (getting a session via SftpRemoteFileTemplate.execute(session -> ...)).

However, I agree that the utility should catch and ignore that exception. Please open an issue on GitHub.

Contributions are welcome.