I am hoping someone has got this to work. I am the trying to connect RabbitMQ shovel to Azure Event hub or Service Bus. I found two tutorials on MS, (very short to set up) but both results in the shovel error. The request to Event hub or Service bus is success in the example.
My guess is that something is missing from the below tutorials or that some components have been changed and breaking changes has been introduced:
Importing Data from RabbitMQ into Azure Data Explorer via Event Hubs https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/importing-data-from-rabbitmq-into-azure-data-explorer-via-event/ba-p/3777688 Shovel error
AMQP allows for interconnecting connecting brokers directly, for instance using routers like Apache Qpid Dispatch Router or broker-native "shovels" like the one of RabbitMQ. https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-overview
How to integrate Service Bus with RabbitMQ https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-integrate-with-rabbitmq
Version: RabbitMQ 3.10.7, Erlang 25.0
Case and steps: Adding a new Azure Service Bus Namespace rabbitmq01 basic public endpoint Creating our Azure Service Bus Queue from-rabbitmq01
Enabling the RabbitMQ Shovel Plugin
rabbitmq-plugins enable rabbitmq_shovel_management
Created queue in Rabbitmq telemetry01
Connecting RabbitMQ to Azure Service Bus Add SAS Policy, Manage now, tested with just send also. rabbitmq-shovel01 Primary Connection String
Connection String to AMQP https://red-mushroom-0f7446a0f.azurestaticapps.net/ amqps://rabbitmq-shovel01:[email protected]:5671/?sasl=plain
TNC
Test-NetConnection -ComputerName rabbitmq01.servicebus.windows.net -Port 5671 = True
Make the dynamic shovel with above properties. Shovel was done exactly like the tutorial The shovel status is just starting, never goes to running.
Log:
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> supervisor: {<0.742.0>,amqp10_client_connection_sup} 2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> errorContext: start_error 2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> reason: {badmatch,{error,closed}}
[...]
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** Stacktrace = 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** [{amqp10_client_frame_reader,init,1, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> [{file,"amqp10_client_frame_reader.erl"}, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {line,109}]}, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {gen_statem,init_it,6,[{file,"gen_statem.erl"},{line,1001}]}, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]
[...]
2023-07-08 19:09:56.386000+02:00 [error] <0.688.0> Shovel 'rmq-2-sb' could not connect to destination 2023-07-08 19:09:56.387000+02:00 [error] <0.687.0> supervisor: {<0.687.0>,rabbit_shovel_dyn_worker_sup}
Shovel to Bus requests success
Tested
I have sent a mail to RabbitMQ community also. Thank you in advance
11.07.2023 Update New RabbitMQ 3.12.1 and Erlang 26.0
Today I did a new test with both tutorials. For the Event hub i followed it exactly. The shovel error is:
2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> supervisor: {<0.2006.0>,amqp10_client_connection_sup}
2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> errorContext: start_error
2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[...]
2023-07-11 10:09:27.788000+02:00 [error] <0.2476.0> Shovel 'shovel-07' could not connect to destination
For the Service bus it is the same.
First I tried with shovel Source AMQP 0.9.1, Destination, AMQP 1.0.
:
2023-07-11 10:44:41.042000+02:00 [error] <0.3627.0> supervisor: {<0.3627.0>,amqp10_client_connection_sup}
2023-07-11 10:44:41.042000+02:00 [error] <0.3627.0> errorContext: start_error
2023-07-11 10:44:41.042000+02:00 [error] <0.3627.0> reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[...]
2023-07-11 10:44:41.043000+02:00 [error] <0.3610.0> Shovel 'shovel_01' could not connect to destination
Then I tried shovel Source AMQP 1.0, Destination, AMQP 1.0.
And as expected:
2023-07-11 11:08:18.155000+02:00 [error] <0.8999.0> amqp1_0_plugin_not_enabled
Enabled plugin amqp1_0
2023-07-11 11:14:58.438000+02:00 [error] <0.9613.0> supervisor: {<0.9613.0>,amqp10_client_connection_sup}
2023-07-11 11:14:58.438000+02:00 [error] <0.9613.0> errorContext: start_error
2023-07-11 11:14:58.438000+02:00 [error] <0.9613.0> reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[...]
2023-07-11 11:14:58.438000+02:00 [error] <0.9582.0> Shovel 'shovel_03' could not connect to destination
After update to: RabbitMQ 3.12.1 and Erlang 26.0 .So it has do to with TLS, and that was not a step in any of the tutorials. This is gonna take time :) Any help is appreciated
I got in touch with MS, as per delivered test results, they did not find a issue with the test(s) done. TNC images validated connectivity. Further troubleshooting will be done by support. Looking forward to that. Collaboration is key.
https://www.erlang.org/blog/otp-26-highlights/ Erlang/OTP 26 Highlights May 16, 2023 [...] Erlang/OTP 26 Highlights In OTP 26, the default value for the verify option is now verify_peer instead of verify_none. Host verification requires trusted CA certificates to be supplied using one of the options cacerts or cacertsfile. Therefore, a connection attempt with an empty option list will fail in OTP 26:
{error,{options,incompatible,
[{verify,verify_peer},{cacerts,undefined}]}}
The default value for the cacerts option is undefined, which is not compatible with the {verify,verify_peer} option. To make the connection succeed, the recommended way is to use the cacerts option to supply CA certificates to be used for verifying.
This is the breaking change (I would believe) after 16.05.2023:
amqps://rabbitmq-shovel:[email protected]:5671/?sasl=plain
Is must be on format similar to this I would belive:
amqps://rabbitmq-shovel:[email protected]:5671/?cacertfile=/path/to/cacert_root_certificate_servicebus.pem&verify=verify_peer (something??)
So https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-integrate-with-rabbitmq must be updated with new information and the tool https://red-mushroom-0f7446a0f.azurestaticapps.net/ must also generate new URI with ssl stuff, and I would belive the public root certificate for Azure Service Bus must be obtainable. Input in this?
15.07.2023 update https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/importing-data-from-rabbitmq-into-azure-data-explorer-via-event/ba-p/3777688 Tutorial version: RabbitMQ 3.11.7 Erlang 25.2.
Tested with RabbitMQ 3.11.10 Erlang 25.2.
RabbitMQ log:
2023-07-15 10:38:00.787000+02:00 [warning] <0.1427.0> Description: "Server authenticity is not verified since certificate path validation is not enabled"
2023-07-15 10:38:00.787000+02:00 [warning] <0.1427.0> Reason: "The option {verify, verify_peer} and one of the options 'cacertfile' or 'cacerts' are required to enable this."
2023-07-15 10:38:00.787000+02:00 [warning] <0.1427.0>
2023-07-15 10:38:01.394000+02:00 [error] <0.1408.0> Shovel 'rmq2eh' in virtual host '/' is stopping, reason: {outbound_link_detached,
2023-07-15 10:38:01.394000+02:00 [error] <0.1408.0> {'v1_0.error',
2023-07-15 10:38:01.394000+02:00 [error] <0.1408.0> {symbol,
2023-07-15 10:38:01.394000+02:00 [error] <0.1408.0> <<"amqp:unauthorized-access">>},
[...] 2023-07-15 10:49:09.143000+02:00 hostname =>
2023-07-15 10:49:09.143000+02:00 <<"xxxxxxxx.servicebus.windows.net">>,
The tutorial(s) must be update with highest RabbitMQ and Erlang version it supports, so that new user know what version(s) they can use and what are the dependency's.. I will do one test with same RabbitMQ version used in the hub tutorial if I can find it. The Erlang version used for this test is the same.
Update versions (could not find 3.11.7): RabbitMQ 3.11.0 Erlang 25.2 result:
2023-07-15 11:46:16.886000+02:00 [error] <0.2202.0> Shovel 'rmq2ev114' in virtual host '/' is stopping, reason: {outbound_link_detached,
2023-07-15 11:46:16.886000+02:00 [error] <0.2202.0> {'v1_0.error',
2023-07-15 11:46:16.886000+02:00 [error] <0.2202.0> {symbol,
2023-07-15 11:46:16.886000+02:00 [error] <0.2202.0> <<"amqp:unauthorized-access">>},
[...]
2023-07-15 11:46:16.887000+02:00 [warning] <0.2220.0> hostname =>
2023-07-15 11:46:16.887000+02:00 [warning] <0.2220.0> <<"xxxxx.servicebus.windows.net">>,
[...]
2023-07-15 11:46:21.895000+02:00 [warning] <0.2254.0> Description: "Server authenticity is not verified since certificate path validation is not enabled"
2023-07-15 11:46:21.895000+02:00 [warning] <0.2254.0> Reason: "The option {verify, verify_peer} and one of the options 'cacertfile' or 'cacerts' are required to enable this."
15.07.2023 18:11 It works now finally. It now works for Azure Event Hub and for Azure Service bus, new URI with TLS and Azure Root certificate and had to add an extra root policy also to the namespace for each service, not just the sub SAS https://github.com/spawnmarvel/test/blob/main/it_works.jpg
I am out, I am done, my life is back.
