What would be an easy and secure way to push data into Azure Data Lake Gen2?
What would be the simple and still secure way to enable someone from outside the organization (source system) to push file (CSV) with the data into the organization's Data Lake Gen2? So far it was a CSV file pushed into the FTP server, now the data needs to end up in Data Lake.
Is there any Azure Service that could be exposed? Or does the source need to install CLI and set up a user to connect with?
I'm collecting the options to see if there is one secure enough, and yet simple so that it won't require the source to install or set up any sophisticated software. The more similar to the simple (S)FTP transfer it would be, the better.
One interesting and more recent option would be to configure your ADLS Gen2 account for SFTP. Externally, the user can use standard SFTP mechanisms to push the data to your ADLS account. Internally, you only need to read from the storage account, no SFTP required.
From a security standpoint, when you create an SFTP user, it is unique to that storage account. You can select how the SFTP can connect:
Further, you define the container(s) the account can access and the container level permissions:
We've done this for several customers so far and have been happy with the results.