Search code examples
azure-eventhubazure-data-explorerkql

Can we inject compressed data to Kusto in plain text via Event Hub


I am trying to ingest payload to Kusto/ADX via Event Hub

Limitation = 1 standard tier EH can only support throughput up to 40 Mbps.

Goal: Increasing max throughput by sending compressed payload without handling translation manually.

Example: payload = { a: 1, b: 2 }

we send this payload to EH by compression manually and Kusto store it as 1 row with 2 column a and b. Not handling compression handle from our end.

I am expecting Event hub to handle the compressed data and translation from their end.


Solution

    1. It's 40 MB (Megabyte) per second, not 40Mb (Megabit).

    2. You can compress your payload with gzip.
      Kusto will open it automatically as part of the ingestion process.

      Ingest data from event hub into Azure Data Explorer

      Setting Suggested value Field description
      Compression None The compression type of the event hub messages payload. Supported compression types: None, Gzip.

    Having said that, the right thing to do would probably be to switch to an Event Hubs in higher tier.