Finalizer thread are responsible clearing objects in the finalization queue. Ironically does the same finalizer thread can responsible for OOM?
The short answer is yes.
Some classes implement the Object.finalize() method. Objects which override this method need to called by a background thread call finalizer, and they can't be cleaned up until this happens. If these tasks are short and you don't discard many of these it all works well. However if you are creating lots of these objects and/or their finalisers take a long time, the queue of objects to be finalised builds up. It is possible for this queue to use up all the memory.
If many objects with finalizers are created, it is likely that there will be performance issues, even if the underlying native resources are explicitly freed using try-finalize blocks.
Code to try and outpace java garbage collector with finalizers showed the following result.
It is surprisingly easy to outpace the Java garbage collector if lots of objects with finalizers are created, leading to spurious out-of-memory errors with lots of objects which could theoretically be reclaimed.More on this can be found on the link http://www.enyo.de/fw/notes/java-gc-finalizers.html
There are some apps that have hit this finalizer queue build up problem in the past, so it is worth considering how to deal with it. One obvious way is to increase the priority of the "Finalizer" daemon thread - there is no API for this, so you have to run through all the threads to find it by name, then increase it's priority.
You could also take explicit control over finalization by removing the finalize() method and using your own explicit queue using your own Reference objects in a very similar way that the Finalizer class processes the objects and their finalize() methods . That way you control your finalization processing thread's priority and schedule.
Note that neither of these techniques reduce the overheads in having finalizable objects, they just avoid the queue building up because of the lower priority thread.
How finalization works is shown in the link below for the curious