How to use librdkafka with OIDC and Azure AD as token provider for OAUTHBEARER?

Question

Problem

I want to use Kafka OIDC with Azure AD as a token provider, but I'm stuck on some strange errors.

I have read many azure docs pages and tried their examples from https://github.com/Azure/azure-event-hubs-for-kafka - all those worked for me when I tested with python and C++. Including oauth(which uses Azure AD) examples listed in https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth/python

Errors

When I attempted to use the credentials from the registered app's webpage following the https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-protocols-oidc and using the v2 endpoint, I ended up with ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: sasl_ssl://myhub.servicebus.windows.net:9093/bootstrap: SASL authentication error: Invalid URI: The format of the URI could not be determined. (after 272ms in state AUTH_REQ). For the v1 without the scope, the error is similar - FAIL|rdkafka#producer-1| [thrd:sasl_ssl://myhub.servicebus.windows.net:9093/bootstrap]: sasl_ssl://myhub.servicebus.windows.net:9093/bootstrap: SASL authentication error: Invalid URI: The format of the URI could not be determined. (after 259ms in state AUTH_REQ).

Setup

Again, all the existing examples I tested work for me, but none of those uses OIDC. The configuration I'm using is below - what should I use in sasl.oauthbearer.token.endpoint.url, sasl.oauthbearer.config to make this work?

    conf->set("bootstrap.servers", "myhub.servicebus.windows.net:9093", errstr);
    conf->set("security.protocol", "SASL_SSL", errstr);
    conf->set("sasl.mechanism", "OAUTHBEARER", errstr);
    conf->set("sasl.oauthbearer.method", "oidc", errstr);
    conf->set("sasl.oauthbearer.client.id", "***", errstr);
    conf->set("sasl.oauthbearer.client.secret", "***", errstr);
    conf->set("sasl.oauthbearer.token.endpoint.url", "https://login.microsoftonline.com/***/oauth2/v2.0/token", errstr);
    conf->set("sasl.oauthbearer.scope", "api://***/.default", errstr);

Links

Confluent Kafka OIDC - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186877575

C++ librdkafka OIDC - https://github.com/edenhill/librdkafka/blob/b871fdabab84b2ea1be3866a2ded4def7e31b006/src/rdkafka_sasl_oauthbearer_oidc.c#L242

Github issues question - https://github.com/Azure/azure-event-hubs-for-kafka/issues/223

Edited:

Further debugging with debug=all revealed that OAUTHBEARER token is received while the app fails on SASL Authentication Sequence.

SASL OAUTHBEARER client in state client-first-message
Send SASL Kafka frame to broker
Sent SaslAuthenticateRequest

Received SaslAuthenticateResponse
Invalid URI: The format of the URI could not be determined.

Answer 1

Okay, Azure responded that Kafka OIDC isn't tested on their side and may not be supported by the broker.

But I tested that it works with Kafka OIDC - I will contribute with the examples to their repo soon.

The problem was using scope for the registered app(api://{app-id}/.default) vs scope for the namespace(https://myhub.servicebus.windows.net/.default) - latter one is correct.

That, in turn, led to receiving the incorrect token-type and Invalid URI format - what really helped here was example from the confluent-kafka with no azure dependencies involved https://github.com/confluentinc/confluent-kafka-python/blob/master/examples/oauth_producer.py