Following this guidelines for sse in APIM I'm trying to implement event hub logging to send the response of SSE events - Azure OpenAI Streaming API as I would like to calculate the tokens, in my case there are multiple different clients/users and want to do this calculation on my side(APIM) so that I can tell the number of tokens are consumed by different clients/users.
The current policy will look like below.
<log-to-eventhub logger-id="apim-log-namespace-test-stream">@{return new JObject(
new JProperty("records", new JArray() {
new JObject(
new JProperty("EventTime", DateTime.UtcNow.ToString()),
new JProperty("ServiceName", context.Deployment.ServiceName),
new JProperty("RequestId", context.RequestId),
new JProperty("RequestIp", context.Request.IpAddress),
new JProperty("OperationName", context.Operation.Name),
new JProperty("response", context.Response.Body.As<string>(preserveContent: true).ToString())
)
})).ToString();}</log-to-eventhub>
During the debug I could see it took a good amount of time to read the response payload (for sending to event hub)
{
"source": "response-payload-handler",
"timestamp": "2023-10-03T08:03:59.4607171Z",
"elapsed": "00:00:01.9290785",
"data": "Reading response payload because next handler requires access to it."
},
{
"source": "response-payload-handler",
"timestamp": "2023-10-03T08:04:05.1387406Z",
"elapsed": "00:00:07.6014120",
"data": "Reading response payload succeeded. 131316 bytes read."
},
If I remove below line
new JProperty("response", context.Response.Body.As<string>(preserveContent: true).ToString()))
I don't see the repose-payload-handler task that is consuming a lot of time earlier. I also think due to implementing the eventhub for sending the response body, I suspect the streaming is not happening (checked from postman) It wait to give the full response till it completes the eventhub
Question is: Is there any way I could achieve this streaming (SSE Events) to log the response body async somewhere (event hub or servicebus ) without affecting the original streaming capability?
You are right about this breaking streaming and is as such mentioned in the official guide for SSEs.
Unfortunately, I don't believe there is a way to do this in APIM today. Your best bet would be to offload this to an Azure Function that essentially "proxies" the stream while allowing you to inspect chunks and log them.