Search code examples
azureazure-storageairflowazure-files

I'm getting a lot of transactions while idling (Airflow and Azure File Share)

I need to load data from different files into an Azure SQL database. So I set up a VM running Airflow and two Azure File Shares, one for my dags (so that I can modify them without sshing into the VM) and another to drop the files that will be loaded.

I mounted those two fileshares to the VM and my PC and use them as normal drives.

The system is currently idling and I can see in Azure's portal that I'm getting about 24k transactions every 5 minutes, but I can't see specifically what is generating them.

Is it possible the VM is constantly requesting a list of files or touching the fileshare to check if it's still there? How can I avoid this?

Thanks!

Solution

  • I can confirm that having the dags folder in a shared drive was the cause of the insane amount of transactions. I moved the dags folder to the VM drive and now everything is back to normal.