Search code examples
windowsdllmanifestaslr

Forcing Windows to load DLL's at places so that memory is minimally fragmented

My application needs lots of memory and big data structure in order to perform its work. Often the application needs more than 1 GB of memory, and in some cases my customers really need to use the 64-bit version of the application because they have several gigabytes of memory.

In the past, I could easily explain to the user that if memory reached 1.6 to 1.7 GB of memory usage, it was 'out of memory' or really close to an 'out of memory' situation, and that they needed to reduce their memory or move to a 64-bit version.

The last year I noticed that often the application only uses about 1 GB before it already runs out of memory. After some investigations it seemed that the cause of this problem is memory fragmentation. I used VMMAP (a SysInternals utility) to look at the memory usage of my application and saw something like this: Address Space Fragmentation

The orange areas are memory allocated by my application. The purple areas are executable code.

As you can see in the bottom halve of the image, the purple areas (which are the DLL's) are loaded at many different addresses, causing my memory to be fragmented. This isn't really a problem if my customer does not have a lot of data, but if my customer has data sets that take more than 1 GB, and a part of the application needs a big block of memory (e.g. 50 MB), it can result in a memory allocation failure, causing my application to crash.

Most of my data structures are STL-based and often don't require big chunks of contiguous memory, but in some cases (e.g. very big strings), it is really needed to have a contiguous block of memory. Unfortunately, it is not always possible to change the code so that it doesn't need such a contiguous block of memory.

The questions are:

  • How can I influence the location where DLL's are loaded in memory, without explicitly using REBASE on all the DLL's on the customer's computer, or without loading all DLL's explicitly.
  • Is there a way to specify load addresses of DLL's in your own application manifest file?
  • Or is there a way to tell Windows (via the manifest file?) to not scatter the DLL's around (I think this scattering is called ASLR).

Of course the best solution is one that I can influence from within my application's manifest file, since I rely on the automatic/dynamic loading of DLL's by Windows.

My application is a mixed mode (managed+unmanaged) application, although the major part of the application is unmanaged.

Anyone suggestions?

Solution

  • First, your virtual address space fragmentation should not necessarily cause the out-of-memory condition. This would be the case if your application had to allocate contiguous memory blocks of the appropriate size. Otherwise the impact of the fragmentation should be minor.

    You say most of your data is "STL-based", but if for example you allocate a huge std::vector you'll need a contiguous memory block.

    AFAIK there is no way to specify the preferred mapping address of the DLL upon its load. So that there are only two options: either rebase it (the DLL file), or implement DLL loading yourself (which is not trivial of course).

    Usually you don't need to rebase the standard Windows API DLLs, they are loaded at your address space very tightly. Fragmentation may arrive from some 3rd-party DLLs (such as windows hooks, antivirus injections and etc.)