Search code examples
azureazure-eventhubazure-data-explorer

Ingesting data into Azure Data Explorer from Azure EventHub through service principal

Initial version ---

I'm able to ingest data from an EventHub directly into my Azure Data Explorer(ADX) database. In my production case, the Eventhub is in a different subscription than the ADX cluster. A service principal is given access to the eventhub through the Data Receiver role. I have added the App(Service principal) as an ingestor to the ADX database. It seems like I need to run a query or write some code to have the ADX databse ingest the data from the event hub using the service principal but my search online hasn't been fruitful. What steps should I follow?

Attempt 2 at explaining the situation---

  1. The EventHub only authorizes a ServicePrincipal to receive the data
  2. The App I created has delegated permissions to ADX
  3. ADX database that will be ingesting the data from the EventHub has a RBAC role of DataIngestor assigned to the App in step 2.
  4. This article - https://learn.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app#register-your-application-with-an-azure-ad-tenant very closely describes what I'm trying to achieve, but with a storage account
  5. In the article referenced above - I need the ADX equivalent of this section (https://learn.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app#client-libraries-for-token-acquisition).
Solution

  • Answer for initial version ---

    Assuming what you're trying to achieve is to ingest data from an Event Hub into Azure Data Explorer, when the event hub and the Azure Data Explorer cluster resources are on different subscription, then that's supported and it's not needed to add any additional permissions as you described.

    Here are instructions for how to ingest data from Event Hub into Azure Data Explorer: https://learn.microsoft.com/en-in/azure/data-explorer/ingest-data-event-hub

    When the Event Hub and Azure Data Explorer cluster are on different subscriptions, you need to make sure both subscriptions are registered for the Microsoft.Kusto resource provider for the ingestion to work.

    Here's an article that would help register for a resource provider through the portal: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-supported-services

    Here's an example for how to register for a resource provider using powershell: https://learn.microsoft.com/en-us/powershell/module/azurerm.resources/register-azurermresourceprovider?view=azurermps-6.13.0

    Answer for second version ---

    That's currently not supported, you have to register your subscription to have Kusto be able to get permissions to read from your event hub into your kusto cluster.