We are testing ActiveMQ Artemis 2.22.0 with clients using the core protocol. The broker is configured to apply paging. We let producers fill up the broker with messages until max-disk-usage stopped all producers.
After we tried connecting consumers, which worked in the first place. However, the brokers disk kept filling until the broker crashed completely.
Now, even after we disconnected all clients manually we see that after restart the broker is extending its message journal until the disk is full again. After restart we see a lot of messages saying deleting orphaned file. The disk usage goes down. After some seconds however, the journal starts rising again and the story repeats.
That's probably not enough information to clearly solve our issue. Thus, here are my questions:
What are possible causes to fill disk space if neither consumers nor producers are connected?
You may be hitting ARTEMIS-3868 in which case I strongly recommend you move to the latest release (i.e. 2.25.0 at this point).
How can we debug such a situation?
The first thing to do would be to use the artemis data print command to print details about exactly what is in the journal. Hopefully that will shed light on what is causing the journal growth.
In case (really not hoping for that), the journal became corrupt. Is there any way first elaborate that and second restore it?
Particular records in the journal may be corrupted and lost, but the broker should be able to read and recover everything that is still viable.