What is approximate worst case garbage collection duration on big heaps

I need a rule of thumb regarding the maximal time of full garbage collection. The motivation is to be able to distinguish between a faulty JVM process and a process which is under GC.

Suppose I have a regular general-purpose server hardware, HotSpot JVM 8, the heap size is 20G-40G, no specific GC and memory options are set. What is a reasonable time limit for GC to complete? Is it 5 minutes, 20 minutes or up to hours?

Update: My application is a memory intensive offline job dealing with big data structures. I don't need to tune GC at all. 10 seconds and 10 minutes pauses are bot fine if this limit is known.

Solution

It is very difficult to quantify how long GC "should" take because it depends on a number of factors:

How big the heap is.
How full the heap is; i.e. the ratio of garbage to non-garbage when you run the GC.
How many pointers / references there are to traverse.
Which GC you are using.
Whether this is minor "new generation" collection, a major "old generation collection" or a "full" collection. The last is typically performed by the fall-back collector when a low-latency collector cannot keep up with the rate of garbage generation.
Whether there is physical <-> virtual memory thrashing occurring.

There are a couple of pathological situations that can cause excessive GC times:

If the heap is nearly full, the GC uses an increasing proportion of time trying to reclaim the last bit of free space.
If you have a heap that is larger than the physical memory available, you can get into virtual memory "thrashing" behavior. This is most pronounced during a major or full GC.

If you do need to pick a number, I suggest that you use one that "feels" right to you, and make it a configuration parameter so that it is easy to tweak. Also, turn on GC logging and look at the typical GC times that are reported there. (Particularly when the server load is high.)