Could you, please, suggest - if I need maximum intersystem/language compatibility, while the main server will be implemented on .NET, could I use NServiceBus or MassTransit, or is it better to use pure RabbitMQ instead? NServiceBus or MassTransit would give pretty nice abstracrion level, but the easiness of communication between different solutions and environments are critical. I'm thinking more of using pure RabbitMQ now, but would be really gratefull for pointing put some pros and cons if I'm taking it into account wrong.
If you have multiple applications with different languages internally that must send and receive messages I would not recommend NServiceBus or MassTransit. They require certain message headers that they themselves add. You would never be able to leverage all the functionality offered by these messaging frameworks. However, if internally you are all .NET and you have multiple applications that will use the messaging infrastructure then NServiceBus and MassTransit will add a lot of value.
Regarding interoperability with third parties. A strength and weakness of both NServiceBus and MassTransit is that you must send and receive messages as strongly typed classes or interfaces. These get serialised to JSON/XML/BSON etc and deserialised back to types again.
Because of this they require message headers that indicate the type of the message for deserialisation purposes. Without the message type headers they won't work.
Working with types is much easier but it can cause interoperability issues. When integrating with third parties that send XML or JSON messages with no types requires you to create a translation tier between your services and the third party.
You could translate to your own type that maps onto the XML/JSON or translate to a simple type that contains a string property for the XML/JSON. Either way, your translation tier would publish the messages internally using MassTransit/NServiceBus and so the messages would contain all the necessary headers to fully exploit all the functionality they offer.
For sending messages to the third party, the translation tier would translate the messages to the XML/JSON expected by the third party.
How much of your messaging system is involved with 3rd party/inter-system integrations? If the answer is very little then NServiceBus and MassTransit would be great options as they offer lots of great functionality.
If the answer is a lot, then they might still be good options. Having a translation tier will shield your internal services from being exposed to the requirements and changing schemas of your third parties. It will give you greater control and flexibility at a cost of extra moving parts.
Ultimately, the translation tier is not as complex as implementing the patterns offered out-of-the-box by NServiceBus and MassTransit. So I would seriously consider them as a viable option.
Some links regarding inter-operability
https://docs.particular.net/nservicebus/messaging/third-party-integration
http://masstransit-project.com/MassTransit/advanced/interoperability.html