Search code examples
azureazure-data-factory

ADF Out of memory exception

I have an ADF with a self-hosted Integrated Runtime and I intermittently get this error:

{
    "errorCode": "2200",
    "message": "'Type=System.OutOfMemoryException
        ,Message=Exception of type 'System.OutOfMemoryException' was thrown.
        ,Source=Microsoft.DataTransfer.ClientLibrary,'",
    "failureType": "UserError",
    "target": "CopyDataEtllongtable",
    "details": []
}

screenshop of IR pipelines

The problem is when I check the Integrated Runtime logs to see where the memory ran out, the graphs are not populated.

I need some ideas about how to approach this.

Solution

  • Are you using an Azure or self-hosted integration runtime? If you're using a self-hosted, I would look at the resource usage on the machine that the self-hosted runtime is installed and running on. I've seen this a lot on self-hosted runtimes that are installed on machines undersized to handle the work that the pipeline is pushing down to them.