So, in two previous posts, I lambasted 3rd party game developers for attempting to allocate greater than 2GB of virtual address space on large memory configurations without setting the large address space aware bit the application header.
I was wrong. Put simply, the graphics subsystem (DXGK) in Vista reserves address space for storing graphics data that may need to be swapped in or out of the GPU’s frame buffer. It does this, by default, regardless of whether the application requests it or not. Traditionally, some games and other 3d applications, have done the same (though in a less sophisticated fashion) to improve state restoration (e.g. resuming game state after losing exclusive control of the framebuffer) and to backfill the GPU’s framebuffer).
The problem arises when both Vista’s graphics subsystem and the game are doing these essentially similar tasks, the allocated virtual address space (though not necessarily the working set) of the process explodes in size, causing the process to eventually attempt malloc past 0×7FFFFFFF the raising an exception if the large address aware bit is not set. My recommendation to set the large address aware bit in the exe header essentially acted as a band-aid as it allowed the application.
The real solution was to patch the DXGK to not statically reserve process virtual address space unless required. This significantly reduces the size of the virtual address space commit. The issue is addressed in http://support.microsoft.com/kb/940105 and a patch for the problem can be obtained from PSS for free (and should be available publicly in the near future).
With that said, if you still run into out of memory errors even after applying KB940105, then you need to set the large address space aware bit (plus it really can’t hurt if you have the physical memory). 2GB of address space is getting a tad small.
