Search code examples
azureazure-blob-storageazure-eventhubazure-data-explorerazure-eventgrid

Azure Event Hub listening to blob changes

I have a use case where I have a streaming pipeline streams data into azure blob storage. I want to pick the data up from the azure blob through event hub, and configure the streaming ingestion into Azure Data Explorer.

I have try to create both event grid (blob storage) on ADX cluster, and the event hub namespace with one event hub pointing to my storage account, but I am not capturing the streaming messages into my event hub. The data format that I am working with is parquet.

Solution

  • Below are detailed step-by-step instructions for creating an Event Hubs data connection for Azure Data Explorer and data up from the azure blob to event hub:

    • Create a storage account and select the events.

    enter image description here

    • Create an Event Subscription by selecting the Filter for Event Types, choosing Endpoint Type as Event Hub, and configuring an endpoint by selecting the event type.enter image description here

    enter image description here

    enter image description here

    • Select Azure Data Explorer cluster and database. Create a cluster and database.

    enter image description here

    • Streaming ingestion must be configured on your Azure Data Explorer cluster.

    Get Data

    • Navigate to the Azure Data Explorer portal.

    • In the portal, go to the Query section.

    • Right-click on the database where you want to ingest the data and select Get data.

      enter image description here

    Source

    • In the Get data window, select the Source tab.

    • Choose Event Hubs as the data source.

    enter image description here

    Configure

    • In the Configure tab, fill in the required fields:
    • Subscription: The subscription ID where the Event Hub resource is located. - Event Hub Namespace: The name identifying your namespace. - Event Hub: The specific event hub you wish to connect. - Consumer Group: The consumer group defined in your event. - Data Connection Name: A unique name for your data connection. - Advanced Filters:
      • Compression: The compression type of the event hub messages payload.
      • Event System Properties: Additional event hub system properties.
      • Event Retrieval Start Date: The date from which existing Event Hubs events should be retrieved.
      • Select Next to proceed.

    Inspect

    • The Inspect tab provides a preview of the data. To complete the ingestion process, select Finish.

    • Optionally, use commands to fetch new data or adjust settings in the Command Viewer and Schema Definition File dropdown.

      Edit Columns: Make adjustments based on your data type.

    Mapping Transformations

    • For certain data formats (Parquet, JSON, and Avro), apply ingest-time transformations in the Edit Columns window. enter image description here

    enter image description here