Help with strange memory behavior. Looking for leaks both in my brain and in my code

I spent the last few days trying to find memory leaks in a program we are developing.

First of all, I tried using some leak detectors. After fixing a few issues, they do not find any leaks any more. However, I am also monitoring my application using perfmon.exe. Performance Monitor reports that 'Private Bytes' and 'Working Set - Private' are steadily rising when the app is used. To me, this suggests that the program is using more and more memory the longer it runs. Internal resources seem to be stable however, so this sounds like leaking to me.

The program is loading a DLL at runtime. I suspect that these leaks or whatever they are occur in that library and get purged when the library is unloade开发者_如何转开发d, hence they won't get picked up by the leak detectors. I used both DevPartner BoundsChecker and Virtual Leak Detector to look for memory leaks. Both supposedly catch leaks in DLLs.

Also, the memory consumption is increasing in steps and those steps roughly, but not exactly, coincide with certain GUI actions I perform in the application. If these were errors in our code, they should get triggered every single time the actions are performed and not just most of the time.

Whenever I am confronted with so much strangeness, I begin to question my basic assumptions. So I turn to you, who know everything, for suggestions. Is there a flaw in my assumptions? Do you have an idea of how to go about troubleshooting a problem like this?

Edit:

I am currently using Microsoft Visual C++ (x86) on Windows 7 64.

Edit2:

I just used IBM Purify to hunt for leaks. First of all, it lists a full 30% of the program as leaked memory. This can not be true. I guess it is identifying the whole DLL as leaked or something like that. However, if I search for new leaks every few actions, it reports leaks that correspond with the size increase reported by Performance Monitor. This could be a lead to a leak. Sadly, I am only using the trial version of Purify, so it won't show me the actual location of those leaks. (These leaks only show up at runtime. When the program exits, there are no leaks whatsoever reported by any tool.)

Monitoring the app's memory use with PerfMon or Task Manager is not a valid way of checking for memory leaks. For example, your runtime may just be holding on to extra memory from the OS for pre-allocation purposes or due to fragmentation.

The trick, in my experience, is the CRT debug heap. You can request information about all live objects, and the CRT provides functions to compare snapshots.

http://msdn.microsoft.com/en-us/library/wc28wkas.aspx

It's difficult to know without seeing your code, but there are less obvious ways that "leaks" can occur in a C++ program, e.g.

1. memory fragmentation - if you are allocating different size objects all the time, then sometimes there won't be a large enough contiguous area of free memory and more will have to be allocated from the OS. Allocations like this will not be freed back to OS until all the memory in the allocation is freed, so long running programs will tend to grow (in terms of address space used) over time.

2. forgetting to have a virtual in a base case which has virtual functions - a very common gotcha which leads to leaks.

3. using smart pointers, such as shared_ptr, and have an object hold on to a shared_ptr to an object - memory leak tools won't usually spot this kind of thing.

4. using smart pointers and getting circular references - you need to use e.g. a weak_ptr somewhere to break the cycle.

As to tools, there is purify which is good but expensive.

Perfmon is fine for letting you know if you're leaking, but it's primitive. There are commercial products that will do much better. I use AQTime for C++ code and it's excellent: http://www.automatedqa.com/products/aqtime/

It will tell you the line of code that allocated the memory that was leaked.

Perfmon looks at the number of (4K) pages allocated to your program. Those will typically managed by the heap manager. For instance, if your button press requires 3 allocations of 1 KB each, the heap manager will have to request a new page the first three times. The fourth time, it still has 3KB left. Therefore, you cannot conclude that your button press must have an externally visible effect every time.

I have a non traditional technique to help find a suspected leak in code, that I've used countless times and it is very effective. Clearly it's not the only or best way to find leaks, but it's a trick you should have in your bag.

Depending the depth of your knowledge of the code, you may have a couple of suspect spots in mind. What I've done in the past is target those suspect spots by (what I call) amplifying the leak. This is done by simply putting a loop around a the suspect spot so it is called not once but many times, usually thousands, but that depends on the size of the underlying allocation. The trick is to know where to put the loop. Generally you want to move up the call stack to a spot where within the loop all allocated memory is expected to be deallocated. At run-time use perfmon to watch private bytes and working set, when it spikes you've found the leak. From that point you can narrow the scope of the loop down the call stack to zero in on the leak.

Consider the following example (lame as it may be):

char* leak()
{
    char* buf = new char[2];
    buf[0] = 'a';
    buf[1] = '\0';
}

char* furtherGetResults()
{
    return leak();
}

std::string getResults(const std::string& request)
{
    return furtherGetResults();
}

bool processRequest(SOCKET client, const std::string& request)
{    
    std::string results;

    results = getResults(request);

    return send(client, results.c_str(), results.length(), 0) == results.length();
}

Its not always easy to find the leak if the code is distributed among separate modules or even in separate dlls. It also hard to find because the leaks is so small, but over time can grow large.

To start you can put the loop around the call getResults():

bool processRequest(SOCKET client, const std::string& request)
{    
    std::string results;

    for (size_t i = 0; i < 1000000; i++) {
        results = getResults(request);
    }

    return send(client, results.c_str(), results.length(), 0) == results.length();
}

If the memory usage spikes then you've got the leak, following this you move down the call stack to getResults(), then to furtherGetResults() and so on until you've nailed it. This example overly simplifies the technique but in production code there is typically a lot more code in each function called and it's more difficult to narrow down.

This option may not always be available, but when it is it finds the problem very quickly.