Env: Windows 7, GeForce GTX 750, CUDA 10.0, cuDNN 7.4
Maven dependencies:
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-cuda-10.0</artifactId>
<version>1.0.0-beta3</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-cuda-10.0</artifactId>
<version>1.0.0-beta3</version>
</dependency>
Every 10 minibatches I'm checking performance on the test test. I used to call net.evaluate(), but that gave me this error:
Exception in thread "AMDSI prefetch thread" java.lang.RuntimeException: java.lang.RuntimeException: Failed to allocate 637074016 bytes from DEVICE [0] memory
at org.deeplearning4j.datasets.iterator.AsyncMultiDataSetIterator$AsyncPrefetchThread.run(AsyncMultiDataSetIterator.java:396)
Caused by: java.lang.RuntimeException: Failed to allocate 637074016 bytes from DEVICE [0] memory
at org.nd4j.jita.memory.CudaMemoryManager.allocate(CudaMemoryManager.java:76)
at org.nd4j.jita.workspace.CudaWorkspace.init(CudaWorkspace.java:88)
at org.nd4j.linalg.memory.abstracts.Nd4jWorkspace.initializeWorkspace(Nd4jWorkspace.java:508)
at org.nd4j.linalg.memory.abstracts.Nd4jWorkspace.close(Nd4jWorkspace.java:651)
at org.deeplearning4j.datasets.iterator.AsyncMultiDataSetIterator$AsyncPrefetchThread.run(AsyncMultiDataSetIterator.java:372)
Then I switched from net.evaluate() to net.output() with training = false and reduced the size of test set from 100 to just 20. This worked without errors. I tried to increase the number of records to 30, it showed this warning, but kept working:
2019-01-12 14:47:44 WARN org.deeplearning4j.nn.layers.BaseCudnnHelper Cannot allocate 300000000 bytes of device memory (CUDA error = 2), proceeding with host memory
I can understand that there is not enough memory on the video card (GeForce GTX 750 Spec shows memory of 1G), but since it can use host memory I increased test set size back to 100, and got a permanent failure with this error:
2019-01-12 14:59:29 WARN org.deeplearning4j.nn.layers.BaseCudnnHelper Cannot allocate 1000000000 bytes of device memory (CUDA error = 2), proceeding with host memory
Exception in thread "main" 2019-01-12 14:59:39 ERROR org.deeplearning4j.util.CrashReportingUtil >>> Out of Memory Exception Detected. Memory crash dump written to: C:\DATA\Projects\dl4j-language-model\dl4j-memory-crash-dump-1547294372940_1.txt
java.lang.OutOfMemoryError: Failed to allocate memory within limits: totalBytes (470M + 7629M) > maxBytes (7851M)
2019-01-12 14:59:39 WARN org.deeplearning4j.util.CrashReportingUtil Memory crash dump reporting can be disabled with CrashUtil.crashDumpsEnabled(false) or using system property -Dorg.deeplearning4j.crash.reporting.enabled=false
at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:580)
at org.deeplearning4j.nn.layers.BaseCudnnHelper$DataCache.<init>(BaseCudnnHelper.java:119)
2019-01-12 14:59:39 WARN org.deeplearning4j.util.CrashReportingUtil Memory crash dump reporting output location can be set with CrashUtil.crashDumpOutputDirectory(File) or using system property -Dorg.deeplearning4j.crash.reporting.directory=<path>
at org.deeplearning4j.nn.layers.recurrent.CudnnLSTMHelper.activate(CudnnLSTMHelper.java:509)
Now, I'm assuming maxBytes (7851M) refers to heap size (JVM runs with -Xmx8G -Xms8G), yet I also output Runtime freeMemory() and totalMemory() and it showed the following right before crashing, which is more than enough of free memory:
2019-01-12 15:29:20 INFO Free memory: 7722607976/8232370176
So my question is, where do totalBytes (470M + 7629M) numbers come from and why required 1G cannot be allocated if there is free memory available inside JVM?
Below is the memory crash report:
Deeplearning4j OOM Exception Encountered for ComputationGraph
Timestamp: 2019-01-12 14:59:32.940
Thread ID 1
Thread Name main
Stack Trace:
java.lang.OutOfMemoryError: Failed to allocate memory within limits: totalBytes (470M + 7629M) > maxBytes (7851M)
at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:580)
at org.deeplearning4j.nn.layers.BaseCudnnHelper$DataCache.<init>(BaseCudnnHelper.java:119)
at org.deeplearning4j.nn.layers.recurrent.CudnnLSTMHelper.activate(CudnnLSTMHelper.java:509)
at org.deeplearning4j.nn.layers.recurrent.LSTMHelpers.activateHelper(LSTMHelpers.java:205)
at org.deeplearning4j.nn.layers.recurrent.LSTM.activateHelper(LSTM.java:163)
at org.deeplearning4j.nn.layers.recurrent.LSTM.activate(LSTM.java:140)
at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doForward(LayerVertex.java:110)
at org.deeplearning4j.nn.graph.ComputationGraph.outputOfLayersDetached(ComputationGraph.java:2316)
at org.deeplearning4j.nn.graph.ComputationGraph.output(ComputationGraph.java:1727)
at org.deeplearning4j.nn.graph.ComputationGraph.output(ComputationGraph.java:1686)
at org.deeplearning4j.nn.graph.ComputationGraph.output(ComputationGraph.java:1672)
at org.lungen.deeplearning.net.predictor.CharacterSequenceValuePredictorNet.testOutputAndScore(CharacterSequenceValuePredictorNet.java:195)
at org.lungen.deeplearning.net.predictor.CharacterSequenceValuePredictorNet.train(CharacterSequenceValuePredictorNet.java:166)
at org.lungen.deeplearning.net.predictor.CharacterSequenceValuePredictorNet.main(CharacterSequenceValuePredictorNet.java:283)
========== Memory Information ==========
----- Version Information -----
Deeplearning4j Version 1.0.0-beta3
Deeplearning4j CUDA deeplearning4j-cuda-10.0
----- System Information -----
Operating System Microsoft Windows 7 SP1
CPU Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
CPU Cores - Physical 4
CPU Cores - Logical 8
Total System Memory 15.97 GB (17144102912)
Number of GPUs Detected 1
Name CC Total Memory Used Memory Free Memory
GeForce GTX 750 5.0 2 GB (2147483648) 1.67 GB (1795002368) 336.15 MB (352481280)
----- ND4J Environment Information -----
Data Type FLOAT
backend CUDA
blas.vendor CUBLAS
os Windows 7
----- Memory Configuration -----
JVM Memory: XMX 7.67 GB (8232370176)
JVM Memory: current 7.67 GB (8232370176)
JavaCPP Memory: Max Bytes 7.67 GB (8232370176)
JavaCPP Memory: Max Physical 15.33 GB (16464740352)
JavaCPP Memory: Current Bytes 470.26 MB (493106209)
JavaCPP Memory: Current Physical 3.35 GB (3601498112)
Periodic GC Enabled true
Periodic GC Frequency 100 ms
----- Workspace Information -----
Workspaces: # for current thread 4
Current thread workspaces:
Name State Size # Cycles
WS_LAYER_WORKING_MEM CLOSED 117.40 MB (123100000) 6802
WS_ALL_LAYERS_ACT CLOSED 19.41 MB (20349840) 2400
WS_LAYER_ACT_0 CLOSED 6.23 MB (6528000) 1601
WS_LAYER_ACT_1 CLOSED 381.47 MB (400000000) 1601
Workspaces total size 524.50 MB (549977840)
Helper Workspaces
CUDNN_WORKSPACE 7.06 MB (7408000)
----- Network Information -----
Network # Parameters 1432106
Parameter Memory 5.46 MB (5728424)
Parameter Gradients Memory 5.46 MB (5728424)
Updater Number of Elements 2862812
Updater Memory 10.92 MB (11451248)
Updater Classes:
org.nd4j.linalg.learning.AdamUpdater
org.nd4j.linalg.learning.NoOpUpdater
Params + Gradient + Updater Memory 16.38 MB (17179672)
Iteration Count 400
Epoch Count 0
Backprop Type TruncatedBPTT
TBPTT Length 50/50
Workspace Mode: Training ENABLED
Workspace Mode: Inference ENABLED
Number of Layers 7
Layer Counts
BatchNormalization 2
DenseLayer 1
LSTM 3
OutputLayer 1
Layer Parameter Breakdown
Idx Name Layer Type Layer # Parameters Layer Parameter Memory
1 lstm-1 LSTM 403000 1.54 MB (1612000)
2 lstm-2 LSTM 501000 1.91 MB (2004000)
3 lstm-3 LSTM 501000 1.91 MB (2004000)
5 norm-1 BatchNormalization 1000 3.91 KB (4000)
6 dense-1 DenseLayer 25100 98.05 KB (100400)
7 norm-2 BatchNormalization 400 1.56 KB (1600)
8 output OutputLayer 606 2.37 KB (2424)
----- Layer Helpers - Memory Use -----
# Layer Name Layer Class Helper Class Total Memory Memory Breakdown
5 norm-1 BatchNormalization CudnnBatchNormalizationHelper 1.95 KB (2000) {meanCache=1000, varCache=1000}
7 norm-2 BatchNormalization CudnnBatchNormalizationHelper 800 B {meanCache=400, varCache=400}
Total Helper Count 2
Helper Count w/ Memory 2
Total Helper Persistent Memory Use 2.73 KB (2800)
----- Network Activations: Inferred Activation Shapes -----
Current Minibatch Size 100
Current Input Shape (Input 0) [100, 152, 2000]
Idx Name Layer Type Activations Type Activations Shape # Elements Memory
0 recurrentInput InputVertex InputTypeRecurrent(152,timeSeriesLength=2000) [100, 152, 2000] 30400000 115.97 MB (121600000)
1 lstm-1 LSTM InputTypeRecurrent(250,timeSeriesLength=2000) [100, 250, 2000] 50000000 190.73 MB (200000000)
2 lstm-2 LSTM InputTypeRecurrent(250,timeSeriesLength=2000) [100, 250, 2000] 50000000 190.73 MB (200000000)
3 lstm-3 LSTM InputTypeRecurrent(250,timeSeriesLength=2000) [100, 250, 2000] 50000000 190.73 MB (200000000)
4 thoughtVector LastTimeStepVertex InputTypeFeedForward(250) [100, 250] 25000 97.66 KB (100000)
5 norm-1 BatchNormalization InputTypeFeedForward(250) [100, 250] 25000 97.66 KB (100000)
6 dense-1 DenseLayer InputTypeFeedForward(100) [100, 100] 10000 39.06 KB (40000)
7 norm-2 BatchNormalization InputTypeFeedForward(100) [100, 100] 10000 39.06 KB (40000)
8 output OutputLayer InputTypeFeedForward(6) [100, 6] 600 2.34 KB (2400)
Total Activations Memory 688.44 MB (721882400)
Total Activation Gradient Memory 688.44 MB (721880000)
----- Network Training Listeners -----
Number of Listeners 3
Listener 0 org.x.deeplearning.listener.ScorePrintListener@7b78ed6a
Listener 1 ScoreIterationListener(10)
Listener 2 org.x.deeplearning.listener.UIStatsListener@6fca5907
So, short explanation to close the issue. ND4J uses off-heap memory, which is basically mapped to GPU memory. So, just as @Samuel Audet pointed out, 7629M referred to off-heap memory which clearly did not fit into GPU memory of my GTX 750.
Final note from the DL4J doc:
Note that if your GPU has < 2g of RAM, it’s probably not usable for deep learning. You should consider using your CPU if this is the case. Typical deep-learning workloads should have 4GB of RAM at minimum. Even that is small. 8GB of RAM on a GPU is recommended for deep learning workloads.