I am tuning the Artemis global-max-size attribute, and I noticed that when I set it to 90% of the JVM max heap, it OOMed in a load test I ran. I know the default is half the max memory available to the JVM, but for a broker with 32GB Xmx, for example, this feels too conservative of a limit. I understand that it likely OOMed in my case set to 90% given that Artemis does require memory to track internal state and queue information, but I'm trying to figure out what a good rule of thumb might be in order to keep us from OOMing but not waste resources. This is especially important as we do not page to disk. So we want to keep GC times low but also FAIL as an address full policy.
The answer to your question really depends on your use-case. For example, if you were using MQTT with hundreds of topics and thousands of subscribers each with a few messages in their subscriptions that would involve hundreds of corresponding addresses and thousands of queues and would have a completely different impact on the heap than a use-case involving a single JMS queue even if the overall number of messages is exactly the same.
My recommendation is to explicitly set the global-max-size to half the max heap and monitor the heap size during your load test. Then you can re-test and bump up the global-max-size incrementally until you find the sweet spot for your use-case.
Also, ensure your load-test closely mimics that kind of load you'll actually have in production.