Analyze the "runnable" thread dump under high load server side

The thread dump from a java based application is easy to get but hard to analyze!

There is something interesting we can see from the thread dump.

Suppose we are in开发者_运维百科 a heavily load java web app. And I often take 10 or 15 thread dump files during the peak time ( under the high loads) to make the wide data. So first, there is no doubt that we need tune up the codes whose status is Blocked and Monitor. I can't dig into more with the rest Runnable threads.

So, if the "method" appears from the thread dump many time, can we say it is slower or faster than the other under high load server side? Of course, we can use more profiling tools to check that but the thread dump may give us the same useful information, especially we are in the production env.

Thank you in adv!

Vance

I would look carefully at the call stack of the thread(s) in each dump, regardless of the thread's state, and ask "What exactly is it doing or waiting for at that point in time, and why?"

You don't just look at functions on the call stack, you look at the lines of code where the functions are called. That tells you the local reason for the call. If you combine the local reasons for the calls on the stack, that gives you the complete reason (the "why chain") for what the thread was doing at that time.

What you're looking for is bad reasons that appear on more than one snapshot. (It only takes one unnecessary call on the stack to make that whole sample improvable, so the deeper the stack, the better the hunting.) Since they're bad, they can be fixed and you will get improved performance. The amount of the performance improvement is roughly the fraction of snapshots that show them. That's this technique.

I'd say that if a method appears very often in a thread dump you'd have to either

1. optimize that method since it is called many times or
2. check whether that method is called too often

If you see that the thread runs spend lots of time in a particluar method, there might also be some bug (like we had with using a special regex that suffered from a bug in the regex engine). So you'd need to investigate that.