Search code examples
file-transfer

Receive file remotely for data analysis


I need to get files with data to run my own program for each file or bunch of files. What is the best way to receive those files?

Do I need to set up my own server that expose API for file upload? Should I use S3 and set up a server that fetch files from it automatically? or Maybe DropBox? Or should I use SFTP server? The amount of files are not big (max 1Giga per day)

I'm using Spring boot for my own server


Solution

  • By using SFTP, you can listen to changes (e.g. file upload) in a remote directory by connecting a listener. SFTP, in my opinion, is a great choice for file integration.

    You said you will be using Spring Boot. Spring has a project called Spring Integration, which has SFTP support.

    Spring Boot has a starter called spring-boot-starter-integration useful for bringing the best suited Spring Integration versions to your Spring Boot project. Spring Integration SFTP is really easy to use, since a lot of abstraction is provided by the framework.