EDIT1: intershop.urlrewrite.CheckSource is turned off already
We are recently having quite big problems with URL rewrite rules not being loaded in test and production multi-node environments. The problem started happening after introducing another organization and it's related application onto the servers. From then on we have tried multiple changes and debugging methods to try to figure it out, but without any result. Also the major problem is that it doesn't happen all the time and server restart can fix it but not always.
Here are the details so far how the problem manifests (this has been going on for more than a month now on our production system):
After setting up more detailed log messages and turning on debug messages for URL rewrite classes we have come to the conclusion that the rule loading fails. We have come to this conclusion because we have added debug message on the very start of our applyExpand() method and it never gets shown. This can be observed on the image below:
All of this leads to conclusion that iterator on line 149 is empty.
Please advise on possible causes of this problem and how to resolve it.
With the help of IS support, we have managed to figure out that the problem was that the URL rewrite rules were in a cartridge that wasn't part of every possible application on server which would result in undefined behavior when loading them (it would load on one appserver and on another it wouldn't).
The fix was to add a new common cartridge for all possible applications which would then hold urlrewrite rules and which would definitely be loaded on server startup.