Flask File upload to Azure Data Lake Store

Question

I am trying to upload a file from a Flask (Flask-restplus) application directly to azure data lake store (gen1).

The flask application is running on azure web app. Is that even possible, or would I need to upload it to the azure web app server first, before moving it to ADLS?

The python library for ADLS (https://github.com/Azure/azure-data-lake-store-python) doesn't seem to have any function for that. For example, ADLUploader expects a local file as source.

Thanks!

Answer 1

No direct way for that.

One way is that as you mentioned, upload to app server, then move to ADLS.

Another possible way is that if you can convert the file content to bytes, then you can use some other methods in ADLS like open() / write(), details like below(just the Pseudocode, you can modify them as per your need):

1.Create a client: myclient = core.AzureDLFileSystem(adlCreds,store_name=adlsAccountName)

2.Get the file name you're uploading, create an empty file in ADLS: myclient.touch("test/myfile.txt")

3.Open the file in ADLS with 'wb' mode: myfile = myclient.open('test/myfile.txt','wb')

4.Use some methods to convert the content of the file you're uploading to bytes

5.Use write() and flush() method to write the bytes content to the files in ADLS:

myfile.write(in_file.read()) #the content should be bytes
myfile.flush(force=True)