Search code examples
azure-functionsazure-iot-hubazure-durable-functionsazure-eventgrid

Aggregate in memory incoming messages from IoT Hub for function app to perform calculations of the resulting batch

I receive into IoT Hub incoming messages from MCU sensors and route these to Event Grid on to a function app (telemetryfunction) which I think keeps the incoming messages in order as when received.

enter image description here

enter image description here

I would like to aggregate these messages received in the function app as a batch of e.g. 5/10 to then run an algorithm on these.

I don't see how I can store on memory the incoming messages received in the function app to then run the algorithms/methods for the accumulated resulting batch.

I would like to avoid using a database and rather hold the data on memory because of the speed at which individual incoming messages are received (10ms). I looked at durable functions but I don't see that would help me store data in memory or queue the messages to create a batch, can someone point me in the right direction please?

Thank you

Solution

  • First of all, Event Grid does not not guarantee ordering.

    You can connect your Function to the Event Hub endpoint of the IoT Hub and use the batch feature. However, you can only specify the max number of items in a batch, not a minimum. This could result in batches of 1-n messages, where n is your maximum.

    Another option is to use Azure Stream Analytics. It supports batching of input events, but all the options are temporal. This means you can't specify how big your batch should be, just how big the timeframe is. This might be a match with your use-case though!