Tips for debugging hard-to-reproduce concurrency bugs?

What are some tips for debugging hard to reproduce concurrency bugs that only happen, say, once every thousand runs of a test? I have one of these and I have no idea how to go about debugging it. I can't put print statements or debugger watche开发者_开发问答s all over the place to observe internal state, because that would change timings and produce overwhelming amounts of information when the bug is not successfully reproduced.

Here is my technique : I generally use a lot of assert() to check the data consistency/validity as often as possible. When one assert fails, the program crashes generating a core file. Then I use a debugger with the core file to understand what thread configuration led to data corruption.

This might not help you but will probably help someone seeing this question in the future.

If you're using a .Net language you can use the CHESS project from Microsoft research. It runs unit tests with every kind of thread interleaving and shows you which ones cause the bug to happen.

There may be a similar tool for the language you're using.

It highly depends on the nature of the problem. Commonly useful are bisection (to narrow down the search space) + code "instrumentation" with assertions for accessing thread IDs, lock/unlock counts, locking order, etc. in the hope that when the problem will reproduce next time the application will either log a verbose message or will core-dump giving you the solution.

One method for finding data corruption caused by concurrency bug:

• Add an atomic counter for that data or buffer.
  • Leave all the existing synchronizing code as is - don't modify them, assuming that you're going to fix the bug in the existing code, whereas the new atomic counter will be removed once the bug is fixed.
• When starting to modify the data, increment the atomic counter. When finished, decrement.
• Core dump as soon as you find that the counter is greater than one (using something similar to InterlockedIncrement)

Targeted unit test code is time-consuming but effective, in my experience.

Narrow down the failing code as much as you can. Write test code that's specific to the apparent culprit code and run it in a debugger for as long as it takes to reproduce the problem.

One of the strategies I use is to simulate interleaving of the threads is by introducing spin waits. The caveat is that you should not utilize the standard spin wait mechanisms for your platform because they will likely introduce memory barriers. If the issue you are trying to troubleshoot is caused by a lack of a memory barrier (because it is difficult to get the barriers correct when using lock-free strategies) then the standard spin wait mechanisms will just mask the problem. Instead, place an empty loop at the points where you want your code to stall for a moment. This can increase the probability of reproducing a concurrency bug, but it is not a magic bullet.

If the bug is a deadlock, simply attaching a debugging tool (like gdb or strace) to the program after the deadlock happens, and observing where each thread is stuck, can often get you enough information to track down the source of the error quickly.

A little chart I've made with some debugging techniques to take in mind in debugging multithreaded code. The chart is growing, please leave comments and tips to be added. http://adec.altervista.org/blog/multithreading-debugging-chart/