After hours upon hours of trying to track down a memory bug in my application, I have managed to reduce it to some very bizarre behaviour in a simple JFX program:
Take the following simple example, which gradually fades a rectangle onto a transparent canvas:
public class Test extends Application {
@Override
public void start(Stage primaryStage) {
primaryStage.initStyle(StageStyle.TRANSPARENT);
int width = 1920;
int height = 1080;
Rectangle rect = new Rectangle(width, height);
rect.setFill(Color.SALMON);
rect.setOpacity(0);
StackPane scenePane = new StackPane();
scenePane.getChildren().add(rect);
primaryStage.setScene(new Scene(scenePane));
primaryStage.setWidth(width);
primaryStage.setHeight(height);
primaryStage.show();
FadeTransition ft = new FadeTransition(Duration.millis(10000), rect);
ft.setToValue(1);
ft.play();
}
public static void main(String[] args) {
launch(args);
}
}
When run with the VM args -Xms100m -Xmx100m, this works no problem at all. However, when I give the VM substantially more memory (such as -Xms1000m -Xmx1000m) it very quickly falls over:
java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at com.sun.prism.impl.BufferUtil.newByteBuffer(BufferUtil.java:90)
at com.sun.prism.impl.BufferUtil.newIntBuffer(BufferUtil.java:121)
at com.sun.javafx.tk.quantum.UploadingPainter.run(UploadingPainter.java:148)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at com.sun.javafx.tk.RenderJob.run(RenderJob.java:58)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
at java.lang.Thread.run(Thread.java:745)
More than one profiler I've tried shows the heap at as taking up barely any of the allocated space - but the process view in task manager shows it shooting through all the available memory in a couple of seconds.
The oddities don't end there however - it only seems to fall over for some values of width and height (a width of 1921 for instance means that the application executes fine, no errors in sight.)
This only seems to occur with a transparent stage. Without the transparent style set on the stage in the first line, all seems well (with all configurations I've tried, anyway.) Likewise, it only occurs on Java 8 (I'm using 8u20) - all is fine with Java 7 / JFX 2.x. I'm running Windows 7x64.
Can anyone reproduce this issue, and could anyone shed any light on what the heck might be going on?! It is by far the most bizarre thing I've encountered in a long while...
UPDATE: I've managed to reproduce this on a separate Windows 8 machine, however another Windows 7 box (and a Mac) both seem fine. Not sure exactly what JFX code path I'm hitting, but it seems entirely machine dependant unfortunately.
This only answers part of your question, but here's why increasing your heap will lead to OutOfMemeoryError:
As you can see from the stacktrace, JavaFX is using a DirectByteBuffer for its processing - so the data is not stored in the heap but in native memory. By increasing your (fixed) heap-size you reduce the amount of free memory your OS can offer as native memory.
E.g. on windows using a 32-bit Java-VM the addressable memory-range is 4GB, 2GB are reserved for the OS leaving 2GB for the java-application. With -Xms1000m -Xmx1000m another GB is reserved for heap leaving 1GB for the VM-code, stack, Non-heap-memory (like PermGen etc.) and what's left of his final GB can be used as native memory.