On our Wildfly 10 applicationserver we are using groovy for jasperreports and other similiar systems. After some time the jvm (java 8) gets an "Out of Memory: metaspace" error. The same issue happens on some older jboss 4 servers with permgen (java 6). We upgraded jasperreports from 3.7.2 to 6.3.1 and the problem still exists. The size of permgen/metaspace is big enough (Max 512 MB).
Our other applications that do not use jasperreports and groovy do not have this problem.
After some research i recognized that this is possibly a memory leak in a classloader (stackoverflow question). It seems that the gc does not unload classes generated from groovy.
On jaspersoft documentation they write, that u should set -XX:+UseConcMarkSweepGC and -XX:+CMSClassUnloadingEnabled as VM Parameter to enable class unloading.
So we changed it but the problem still exists. Does Java 8 still support these Parameters? If not is there any other solution. Currently we have to restart the server every night to avoid this problem.
As Durandal mentioned mentioned in the comments of the question, there where a lot of "sun/reflect/DelegatingClassLoader" and "org/codehaus/groovy/runtime/callsite/CallSiteClassLoader" entries in the heap dump. It seems that groovy has some memoryleak issues. We changed the script enginge from groovy to java for jasper reports and javascript for our other systems. This solved the Problem. Thanks for the help.