Allocate memory error in dll called by Java JNA

I try to integrate DLL using JNA in Java. DLL allocates a lot of memory (~600MB). Library is compiled to 32 bit version, so I need to use 32 bit JVM.

I've got fatal error from JVM

 EXCEPTION_UNCAUGHT_CXX_EXCEPTION (0xe06d7363) at pc=0x772bd928, pid=7976, tid=2444

with detailed information

siginfo: ExceptionCode=0xe06d7363, ExceptionInformation=0x19930520 0x02dce184 0x6a3853d8

and stack

Stack: [0x023d0000,0x02dd0000],  sp=0x02dce0d0,  free space=10232k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [KERNELBASE.dll+0xbd928]
C  [MSVCR120D.dll+0x120b86]
C  [MSVCR120D.dll+0x12c2a1]
C  [face_analysis_libD.dll+0x1157dc]  operator new[]+0xc
C  [face_analysis_libD.dll+0xea995]  Mat<float>::setSize+0xa5
C  [face_analysis_libD.dll+0xe904e]  Recognizer::initialize+0x9e
C  [face_analysis_libD.dll+0xb2bf9]  Analyser::initializeAnalyser+0x1c9

Application is started with params

-Xmx1300m -Xms1300m -Xss10m -XX:MaxDirectMemorySize=3g -XX:+PrintHeapAtGC

Probably the problem is with some memory limits on JVM (library works great with C#). Maybe someone has similar problem and can help mi find solution how to run this library in 32 bit JVM.

Solution

32bit applications are generally restricted to 2GB of virtual address space. Assuming you're on a 64bit windows this can be extended to 4GB if the binary has been built with the largeaddressaware flag. I think 32bit java.exe has not, although I haven't tested it myself.

So if the JVM is already eating 1.3G for the heap, additional memory for thread stacks and other non-heap data structures then there is fairly little room left for the dll to make allocations. If it then needs to make a large contiguous allocation this is probably going to fail and throw a C++ exception.

Your options likely are:

reduce heap size
get a 64bit version of that library instead
see if marking the binary as largeaddressaware helps, there are tools to do that