Nifi: Consume Azure Event Hub with an enterprise proxy environment

I have a use case in Nifi to consume data from Event Hub. However, the Nifi server is sitting behind a proxy. The proxy also requires username and password for authorization. So far, I have found out GetAzureEventHub processor does the job but it lacks full proxy controller service.

I would like to know what could be the best way to deal in such a situation.

Solution

There are 2 ways around it. Azure Event Hubs client, by default, is configured to talk to the Service using an open standard called AMQP.

The IANA assigned port for the secure version of AMQP protocol a.k.a AMQPS is 5671. So, here goes the first option:

You will need to work with your admin to allow traffic on port 5671 and to opt out of username-password (aka BASIC) auth for this traffic. This is a complicated option - but, should be technically possible.

Second option & I guess most practical one - in your case:

configure the Event Hubs library to talk AMQP over WebSockets. While creating EventHubClient, set the TransportType property on ConnectionStringBuilder to AMQP_WEB_SOCKETS. This will allow you to talk on the standard HTTPS port which all proxies are configured to run with - which WebSockets uses. If you configure proxySettings at java process level, the library already understands it. I tried to document the details instructions here.

Now, the only limitation you will hit with this approach is that - apache-nifi uses and older version of EventHubs client (0.14.x), whereas, you will need1.2.0to be able to use theWebSockets over proxy`.

There are some breaking changes between these 2 versions - that we documented here - which should help you with migration. Happy coding!