Visual C++: possible to limit heap size?

I have a problem with an application I'm debugging. Steady state memory usage is a few hundred megabytes. Occasionally (after several hours) it gets into a state where its memory usage soars to many gigabytes. I would like to be able to stop the program as soon as memory usage this happens.

Where control passes through my own code, I can trap excessive memory use with code开发者_JAVA技巧 like this:

bool usingTooMuchMemory()
{
    PROCESS_MEMORY_COUNTERS pmc;
    if (GetProcessMemoryInfo(GetCurrentProcess(), &pmc, sizeof pmc))
        return pmc.WorkingSetSize > 0x80000000u; // 2GB working set
    return false;
}

This doesn't help me because I need to test working set size at the right point. I really want the program to break on the first malloc or new that takes either working set or heap size over some threshold. And ideally I'd like to have this done by the CRT heap itself with minimal overhead because the library likes to allocate huge numbers of small blocks.

The suspect code is in a DLL running in a thread created by my calling code. The DLL links statically to the CRT and has no special heap management. I have source code for the DLL.

Any ideas? Am I missing something obvious?

You can set memory allocation and deallocation hooks, using _CrtSetAllocHook.

You can hook the HeapAlloc function, which malloc calls internally, by using the Detours library.

http://msdn.microsoft.com/en-us/library/aa366778%28v=vs.85%29.aspx

If you clear the IMAGE_FILE_LARGE_ADDRESS_AWARE flag in VS's linker options, the program's heap will be limited to 2GB in size, and should crash if attempts are made to acquire memory that would put it over that limit.