If anyone is working with Azure and could help me, I would appreciate it. I am trying to trigger an action every time the creation of a file is logged in an azure data lake path and I need the following to be true:
It has to be in real time
Should report which file is the one created, at what time and which user
I have tried to do it with the ADL diagnostic logs by routing them to an eventhub but the events arrive normally after 3 minutes, I have also tried to do it with Log Analytics and it takes longer for the information to arrive
For your scenario, you can make use of Azure Event Grids where an event will be fired as soon as a file is created. You can invoke an Azure Function to consume that event and do further processing.
You can find more information about it in the following links:
https://azure.microsoft.com/en-us/blog/event-driven-analytics-with-azure-data-lake-storage-gen2/