I have added the following JVM arguments to Application Server in WebSphere:
-XX:+PrintHeapAtGC
-XX:+HeapDumpOnOutOfMemoryError
What is the way to test if these arguments are working?
The PrintHeapAtGC option prints the heap layout before and after every garbage collection. On a standard WAS configuration it outputs to navite_stdout.log. See this post from IBM on enabling verbose Garbage Collection: http://www-01.ibm.com/support/docview.wss?uid=swg21114927 (Note that in the post it warns you about the large amount of logging that this flag generates)
The HeapDumpOnOutOfMemoryError will generate a java heap dump (.hprof file) when an out of memory occurs. The best way to test this is to decrease the heap size to a very small value (~256MB or so) and if that still doesn't generate the error, you could write a small app that leaks memory. See this post from IBM on getting heapdumps in Solaris: http://www-01.ibm.com/support/docview.wss?uid=swg21242314 . In a standard WebSphere configuration heapdumps are written to the profile root directory (e.g. /opt/WebSphere/AppServer/profiles/profile01)
I'm assuming your running WebSphere on Solaris or HP-UX, since those JVM options are applicable to Sun/Oracle JVM.