Completely reworked due to new information:
Initial problem: JBoss killed itself with "OutOfMemoryError: Metaspace", when redeploying a simple Spring Boot app (2.2.6) multiple times
Using a heap dump I found that the H2 driver (1.4.200) kept a reference and leading to a problem
Afterwards I checked why tomcat does not behave like this, finding
The web application [killerApp] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
So it just works in a tomcat, because it is smart enough to notice the memory leak and preventing it. The JBoss does not do it and runs into trouble...
It looks like a standard spring boot app with H2 is misbehaving regarding the handling of unregistering the H2 driver...at least this is my take away.
I additionally found this one: GitHub - Spring Boot - Discussion about unregister managed JDBC drivers
Now I am unsure, who is at "fault"?
Best and many thanks
Another weekend with MAT & OOMs, I now fixed the memory leaks.
First I now manually deregister the JDBC driver initiated from Spring Boot in the ContextListener (from 2.3.0 Spring boot will deregister automatically -> https://github.com/spring-projects/spring-boot/issues/21221). This solves the first memory leak.
The other comes from io.github.classgraph. The destroy thing is added as an application shutdown hook, which is only executed when the JVM is shut down. -> not a good idea for a ever running application server. They fixed it, but springdoc-openapi-ui / org.webjars:webjars-locator-core are still using the old version with the bug. So I manually increased the classgraph version. This solves the second memory leak...damn I never wanted to know so much about this stuff (otherwise I would program in C :D)
Problem solved :)