Search code examples
azureeventsazure-eventhubazure-data-explorer

How to send data from multiple event hubs to ADX?

I have my service deployed in multiple regions in azure. In all the regions I have an event hub and ADX cluster(database and table). Suppose my service is in two regions X and Y. When user request comes it is processed by the nearest region. The request also contains one property of origin-region. So if user is present in region X and his request contains the property "origin-region" set as region Y, this request will be processed by region X's service and then the data of it will be sent to region X's event hub. Finally I want this data from region X's event hub to Y's ADX table.

What should be the best design of this use case?

I was thinking about creating multiple tables (lets say region-x-table, region-y-table and so on) in each of the different regions. And there will be one to many mapping from event hub to adx table of all the regions. Basically, region-x-table of any region will ingest the data present in the region X's event hub. And the data from X's event hub will be only ingested in that region's ADX table which is present in the request's property "orgin-region".

Region X's event hub's data will be ingested to either region X's ADX table(region-x-table) or region Y's ADX table(region-x-table) based on the value of "origin-region" of the request.

So in each region there will be 1 event hub, and n tables(region-x-table, region-y-table,....) for each region. Is there any better approach to this?

I also thought of creating only one table in ADX and ingest data from multiple region's event hub by checking the data schema but it is not possible as one table can only ingest data from one event hub.

Solution

  • Sending data from multiple event hubs to ADX can be done by creating a data connection for each event hub and configuring the connection settings to event hub, event hub namespace and consumer groups.

    enter image description here

    enter image description here

    • Creating multiple tables in each region and ingesting data from the corresponding event hub into the respective table is also a good approach.
    • It also allows more flexibility for more regions.

    And this will ensure that the data is ingested into the correct table based on the origin-region property of the request.

    Another approach

    Using Azure Stream Analytics to process the data from the event hubs and send it to the appropriate ADX table based on the origin-region property.

    This involves in creation of Stream Analytics job for each region and configuring it to read from the corresponding event hub and write to the appropriate ADX table.

    For more information refer to Create an Event Hubs data connection and Use one-click ingestion data from Event Hub into Azure Synapse Data Explorer.