Search code examples
python-3.xazureazure-data-factoryazure-databricks

Continuous running job in Azure


I am have created a simple Python job to watch a remote SFTP directory for any new files dropped in there by an external system. The task is to basically read the new files dropped into the directory in near real-time basis. The job is currently scheduled in Task Scheduler in a Windows Virtual Machine but I would like to move it to any PaaS Service in Azure to schedule this job to run this continuously but not on batch mode. Can someone let me know if any services in Azure that supports this use case?


Solution

  • You could get an idea of Azure Logic App and SFTP connector which supports monitor, create, send, and receive files on a SFTP server.

    SFTP triggers work by polling the SFTP file system and looking for any file that was changed since the last poll.More details,please see this link.

    BTW,the SFTP connector has limitation that it handles only files that are 50 MB or smaller and doesn't support message chunking.For larger files, you have to use the SFTP-SSH connector.