Why can the private size of your process be so much bigger than the number of allocations you've made?

I'm trying to understand the memory usage of a program I maintain. I've realised that from a process dump I can use windbg to do "heap -a 0" to print every single allocation that my process hasn't yet deleted. Each allocation looks a bit like:

0000000006a905c0: 000a0 . 000a0 [07] - busy (68), tail fill - unable to read heap entry extra at 0000000006a90650

Where the 68 is what I malloced and 000a0 is how much memory is actually used (since there's a bit of overhead).

If I add all these allocations up for my program, I get 34Mb, however, this seems to bear little relationship to the memory summary in !address -summary which shows:

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    132      7ff`e905b000 (   8.000 Tb)          100.00%
<unclassified>                         1234        0`11025000 ( 272.145 Mb)  74.04%    0.00%
Image                                   430        0`03fe6000 (  63.898 Mb)  17.38%    0.00%
Stack                                    93        0`01f00000 (  31.000 Mb)   8.43%    0.00%
TEB                                      31        0`0003e000 ( 248.000 kb)   0.07%    0.00%
NlsTables                                 1        0`00024000 ( 144.000 kb)   0.04%    0.00%
ActivationContextData                    22        0`0001e000 ( 120.000 kb)   0.03%    0.00%
CsrSharedMemory                           1        0`00009000 (  36.000 kb)   0.01%    0.00%
PEB                                       1        0`00001000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                            1340        0`1167d000 ( 278.488 Mb)  75.76%    0.00%
MEM_IMAGE                               430        0`03fe6000 (  63.898 Mb)  17.38%    0.00%
MEM_MAPPED                               43        0`01932000 (  25.195 Mb)   6.85%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                132      7ff`e905b000 (   8.000 Tb)          100.00%
MEM_RESERVE                             631        0`0e970000 ( 233.438 Mb)  63.51%    0.00%
MEM_COMMIT                             1182        0`08625000 ( 134.145 Mb)  36.49%    0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          788        0`04428000 (  68.156 Mb)  18.54%    0.00%
PAGE_EXECUTE_READ                        85        0`027b8000 (  39.719 Mb)  10.81%    0.00%
PAGE_READONLY                           225        0`01984000 (  25.516 Mb)   6.94%    0.00%
PAGE_WRITECOPY                           51        0`00081000 ( 516.000 kb)   0.14%    0.00%
PAGE_READWRITE|PAGE_GUARD                31        0`0003e000 ( 248.000 kb)   0.07%    0.00%
PAGE_EXECUTE_READWRITE          开发者_开发技巧          2        0`00002000 (   8.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                      1`80014000      7fd`cb33c000 (   7.991 Tb)
<unclassified>                            0`0eae9000        0`037d7000 (  55.840 Mb)
Image                                   7ff`7f56e000        0`0062e000 (   6.180 Mb)
Stack                                     0`04120000        0`000fc000 (1008.000 kb)
TEB                                     7ff`fff7c000        0`00002000 (   8.000 kb)
NlsTables                               7ff`fffb0000        0`00024000 ( 144.000 kb)
ActivationContextData                     0`00130000        0`00005000 (  20.000 kb)
CsrSharedMemory                           0`7efe0000        0`00009000 (  36.000 kb)
PEB                                     7ff`fffdf000        0`00001000 (   4.000 kb)

These figures are slightly confusing, because it seems like only memory which is both MEM_PRIVATE and MEM_COMMIT has really been allocated by my program... The rest is in shared DLLs, and other areas. The lead me to try and count up all the MEM_PRIVATE,MEM_COMMIT segments by doing:

!address -f:VAR,MEM_PRIVATE,MEM_COMMIT

This gives output for each segment rather than each malloc... Here's a sample:

   0`12bf0000        0`12bf1000        0`00001000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified> 
   0`12cf0000        0`12cf5000        0`00005000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified> 
   0`12e70000        0`12e75000        0`00005000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified> 
   0`37f20000        0`37f21000        0`00001000 MEM_PRIVATE MEM_COMMIT  PAGE_EXECUTE_READWRITE             <unclassified> 
   0`7ffe0000        0`7ffe1000        0`00001000 MEM_PRIVATE MEM_COMMIT  PAGE_READONLY                      <unclassified>

The 3rd column is segment size and if I add all these up for my program I get a total of 68Mb (Roughly double the sum total of all the allocations). So how do I rationalise these two figures? What governs the size of these segments and why does the total differ so much from all my allocations? Also, the size of all my allocations apears rather small compared to what appears in win7 task manager (Private working set).... So am I missing some memory use somewhere?

Adding up your allocation will not give you the total memory "private memory" used by the heap.

For start, virtual memory allocations are done in 64KB granularity, the heap may be using even larger blocks. Also private memory includes all memory that is not shared (shareable) with other processes. This includes memory allocated with VirtualAlloc as well as copy-on-write pages.

I think I've reproduced my problem with a small example program now which clearly shows what's going on. I suspect my parser (which uses alot of temporary STL objects backed by the heap) is allocating bigish objects at the same time as I'm doing allocations for much longer term objcts:

Here's an extreme example of what can happen if you do this with comments showing what task manager reflects:

   int _tmain(int argc, _TCHAR* argv[])
   {
    void * large_allocs[1024*64];
    void * small_allocs[1024*64];

    prompt("About to allocate memory");
    for (int i = 0; i < _countof(large_allocs); i++)
    {
        AllocateNBytes(&large_allocs[i], 10 * 1024);  
        AllocateNBytes(&small_allocs[i], 1);
    }

        prompt("\Large and small allocations performed");

       //NOTE: Task manager shows 650Mb usage at this point

    for (int i = 0; i < _countof(large_allocs); i++)
        free(large_allocs[i]);
    prompt("Large allocations cleared down, We now only have 64k allocated, but how big is our committed memory?");

        //NOTE: Task manager shows windows taking ages to deal with the fragmented memory, it eventually stops with the memory around 226Mb
        //              for committed and private working set despite now only having 64k of real allocations...  It doesn’t seem to be able to clear down any
        //              further than this...


    for (int i = 0; i < _countof(small_allocs); i++)
        free(small_allocs[i]);
    prompt("Small allocations now cleared down, Notice that the private working set and committed memory take ages to get back down to 0");

        //NOTE: The private working set and committed fall back down to what the program started at but take much longer than if I’d only allocated the 
        //              small allocations in the first place.

    return 0;
    }

    void prompt(char * message)
    {
        printf ("%s\n", message);
        char buf[2];
        printf ("Press any key...\n");
        fgets (buf, 2, stdin);
    }

    void AllocateNBytes( void ** ptr, DWORD sizeptr ) 
    {
        *ptr = malloc ( sizeptr * sizeof(BYTE));
        for (DWORD i = 0; i < sizeptr; i++)
        {
            ((BYTE *) *ptr)[i] = 0;   //Fool the optimiser into thinking we read this
            if (((BYTE *) *ptr)[i] != 0)
            {
                printf("%d", ((BYTE *) *ptr)[i]);
            }
        }
    }

Basically, my program seems to prove that that you can have only a small amount of memmory allocated but have a massive amount commited/reserved by windows which the memmory manager can't work out how to get rid of and can't be used by other programs. I guess the answer is that my short term allocation objects should be getting allocated from a large preallocated pool which can then be thrown away in totality after parse time is over.

No, that's not the right way to go. There's a reason that the heap keeps this memory around and it's because it's much faster to reallocate it. Of course, the OS also allocates whatever memory it wants to allocate. The memory manager chooses to keep it, and it does that because it improves performance.

If your program is not consuming more memory than your system can hold, you should not be looking at the memory usage, mainly because RAM exists to be used and secondly because there are six dozen million implementation details that result in extra memory usage that you will not understand and removing them will not benefit you. Since you're using Windows, the minimum requirements mean that you pretty much must be running on a system which can bite 64MB of memory off fairly easily.

You can use GlobalMemoryStatusEx to determine how much virtual memory you're using. Task Manager is most definitely not a reliable source of memory profiling information.