I have a render loop that listens for pen tablet input and draws from vertex/index buffers (among other things). The vertex data can grow and when it hits certain levels, DispatchMsg(&msg) encounters this:
Unhandled exception at 0x5DDBDEF0 (msvcr110d.dll) in App.exe: 0xC0000005: Access violation writing location 0x00000000.
The total size of the allocated vertex and index buffers in the scene are around the same levels each time:
Total Vertices count: 10391370
Total Indices count: 41565480
Total Vertices size: 249392880 (bytes)
Total Indices size: 997571520 (bytes)
and in another sampling:
Total Vertices count: 9969300
Total Indices count: 39877200
Total Vert size: 239263200
Total Indices size: 957052800
The vertex and index buffers are both D3D11_USAGE_DEFAULT. The total size allocation in the scene is higher than listed above, but the smaller buffers are frequently released. There is also some padding in the vertex/index buffers at the end.
The message about to be dispatched every time (when the exception is reached) is 581, which I believe may be:
#define WM_POINTERUPDATE 0x0245
I really don't mind the scene rendering slowly, but if I am reaching some maximum memory allocation (video memory?) is there a way for the memory to slowly page to/from maim memory at the cost of speed? I tried disabling the draw() call which invokes the pixel, vertex shaders, and drawing calls but the exception still occurs. I would prefer a speed tradeoff or workaround to an exception.
From the numbers in your question, it seems that you're allocating lots of memory for your vertex and index data (about 1.2GB). If your app is 32-bit (x86, as opposed to x64), it only has access to 2GB of memory, so it might very well be that your process is out of memory.
The GPU has access to part of the system memory (which is quite slow compared to the GPU memory, but better than no memory at all). For NVidia GPUs, for example, the NVidia Control Panel shows it under "System Information" > "Shared system memory"; this is usually half the total amount of RAM in your system. As far as I'm aware, this system memory takes up part of your process' address space.
If your "pretty old" GPU has 512MB of memory, it would need about 768MB of additional system memory in order to satisfy your memory requests. This leaves 1.2GB for your application. I guess that your application tries to allocate 1.2GB of system memory in arrays that will take your geometry data that you want to upload to the GPU. This would just fit, but your application code needs to be somewhere, too. In this hypothetical case, you've already run out of memory. Of course, you can get rid of the system memory arrays once you've uploaded the geometry data to the GPU, but at precisely this moment, the system doesn't have enough memory to create a GPU memory buffer of the size that you requested.
Switching the project to be 64-bit (x64 in Visual Studio) would enable your process to take up vastly more memory (in most cases up to what's available in your system and the pagefile), and resolve your problem. If memory constraints are actually the problem.
Another thing: I noticed that - assuming your numbers are right - you allocate 24 bytes per index for the index buffer, just as much as you allocate for each vertex in the vertex buffer. Is that correct? An index (which is just an offset into the vertex buffer) should not be larger than 4 bytes. Do you really try to allocate 6 times as much GPU memory as you actually need, or is it a mis-calculation in your application?
If you quickly release and then re-create a buffer on the GPU, then D3D might keep the "old" buffer around for a while (at least as long as it's still bound to the GPU pipeline), so it might be out of memory because of that. So it might help to actually unbind the buffers that you're about to release (by calling IASetVertexBuffers(NULL, 0, 0, 0, 0) and IASetIndexBuffer(NULL, DXGI_FORMAT_R16_UINT, 0)).