What could cause my program to not use all cores after a while?

I have written a program that captures and displays video from three video cards. For every frame I spawn a thread that compresses the frame to Jpeg and then puts it in queue for writing to disk. I also have other threads that read from these files and decodes them in their own threads. Usually this works fine, it's a pretty CPU intensive program using about 70-80 percent of all six CPU cores. But after a while the encoding suddenly slows down and the program can't handle the video fast enough and starts dropping frames. If I check the CPU utilization I can see that one core (usually core 5) is not doing much anymore.

When this happens, it doesn't matter if I quit and restart my program. CPU 5 will still have a low utilization and the program starts dropping frames immediately. Deleting all saved video doesn't have any effect either. Restarting the computer is the only thing that helps. Oh, and if I set the affinity of my program to use all but the semi-idling core, it works until the same happens to another core. Here is my setup:

• AMD X6 1055T (Cool & Quiet OFF)
• GA-790FX-UD5 motherboard
• 4Gig RAM unganged 1333Mhz'
• Blackmagic Decklink DUO capture cards (x2)
• Linux - Ubuntu x64 10.10 with kernel 2.6.32.29

My app uses:

• libjpeg-turbo
• posix threads
• decklink api
• Qt
• Written in C/C++
• All libraries linked dynamically

It seems to me like it would be some kind of problem with the way Linux schedules threads on the cores. Or开发者_StackOverflow中文版 is there some way my program can mess up so bad that it doesn't help to restart the program?

Thank you for reading, any and all input is welcome. I'm stuck :)

First of all, make sure it's not your program - maybe you are running into a convoluted concurrency bug, even though it's not all that likely with your program architecture and the fact that restarting the kernel helps. I've found that, usually, a good way is a post-mortem debugging. Compile with debugging symbols, kill the program with -SEGV when it is behaving strangely, and examine the core dump with gdb.

I would try to choose a core round-robin a when new frame processing thread is spawned and pin the thread to this core. Keep statistics on how long it takes for the thread to run. If this in in fact a bug in Linux scheduler - your threads will take roughly the same time to run on any core. If the core is actually busy with something else - your threads pinned to this core will get less CPU time.