Integration Runtime out of memory ADF

Question

I am using data flow activity to convert MongoDB data to SQL. As of now MongoDB/Atlas is not supported as a source in dataflow. I am converting MongoDB data to JSON file in AzureBlob Storage and then using that json file as a source in dataflow.

for a json source file whose size is around/more than 4Gb, whenever I try to import projection, the Azure Integration Runtime is throwing following error. I have changed the core size to 16+16 and cluster type to memory optimized.

Is there any other way to import projection ?

Answer 1

Workaround for this was: Instead of pulling all the data from mongo in a single blob, I pulled small chunks (500MB-1GB each) by using limit and skip option in "Copy Data" Activity.

and stored them in different JSON blobs