Search code examples
spring-batchspring-integration

How one can streamline data to sftp in spring batch writer

I have following issue to implement as spring batch application.

  1. Query external endpoint for data
  2. Apply business logic (mapping)
  3. Write to csv file
  4. Upload those file to sftp server

I expected significant amount of read data. It's not possible to keep all of it in the memory. My question is can and how I can merge step 3 and 4 into one. And the most importantly how one can streamline data directly to sftp server.

Solution

  • The SFTP Outbound Channel Adapter supports payload as an InputStream. So, you an transfer data over that stream: https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-outbound.

    You can use RemoteFileTemplate or SftpSession directly, though in the Spring Batch ItemWriter.