Does and increasing resident memory size in the JVM indicate a memory leak?

I am starting up a JBoss 4.2 server instance with the following command-line options:

-Xms8192m -Xmx8192m -XX:+DisableExplicitGC -XX:MaxPermSize=512m

I have NOT received an OutOfMemoryException but I would not expect the memory usage to increase if things are being GC'd as their references die. The resident memory usage (as measured w开发者_开发知识库ith top) starts at ~ 4.2G and steadily increases over the next few days to a week until it hits 8.4G. I still do not receive an Exception. My concern is that during periods of significant activity (JBoss Messaging processing > 10k mssgs/sec) there are processing lags of ~100-700ms that occur every 6-10 seconds. This also seems to correlate with a rise in resident memory usage. This is occurring on a machine with 2 quad-core processors and 32G of memory.

I will be turning on the additional command-line parameters:

-verbosegc -XX:+PrintGCDetails

But, I would like to know if this seems to be a garbage collection issue, or a memory leak? I have tried to track down a potential memory leak for a couple weeks and have found things with the XML processing that I thought would definitely fix it, but that was a dead-end. Does the resident memory usage climb to meet the value of Xmx regardless of whether there is a leak or not (especially with explicit GC disabled)? Are there some other garbage collection parameters that may help if it is indeed not a leak (such as modifying garbage collector types, survivor ratios, or pause targets)?

I know 100-700ms delays do not seem like much but it has the potential to make a significant difference in this application. Thanks in advance for any help/suggestions you can offer.

The definition of a "leak" in Java differs substantially from the definition of a "leak" in, say, C/C++. In C/C++ you get a "leak" when you malloc or new a piece of storage and never subsequently free or delete it.

But in Java, of course, you never delete anything, but leave it up to GC to find stuff that is no longer referenced and free it up.

What can happen, however, is that some complex data structure gets built up, and then built up some more, and then built up some more, sometimes unintentionally.

The most obvious case would be something like a StringBuffer that is used to accumulate log info and never written/emptied. But you can also have a (intentionally) long-lived structure in your application where you happen to "park" some (supposedly) short-lived object, but then fail to null the pointer to the "short-lived" object after you're done with it, causing it to effectively become immortal. Some of these things are obvious, some require considerable investigation to figure out.

But in a large server you also tend to have a fairly steady build-up of "stuff" over a period of time, even without such leaks. Eg, if a given application is called on, it may cause some objects to be created (or just classes loaded, with objects they indirectly create), and those objects may "hang around" until the next time the application is called. Things like web page caches fill up. If something like JSP is used, objects for that will be created and "cached" for later use.

But this build-up of "stuff" should observe an asymptotic behavior, slowly approaching some steady-state value over time. If it continues upward at a steady state, then you probably DO have a "leak".

Re your GC behavior, it's not unusual to have GC running every few seconds on a busy server. You can play with the tuning parameters to try to "balance" the different GC "tiers", but it's a bit of a black art to do so. And often poor GC performance on a server simply a matter of having a GC implementation that isn't well-designed for servers -- GC on a busy server needs to be capable of running in a largely concurrent fashion, and most GC implementations do not do that very well.

I have NOT received an OutOfMemoryException but I would not expect the memory usage to increase if things are being GC'd as their references die.

This is not necessarily correct. It's possible for objects to be placed in tenure memory of the heap. This occurs if the object is referenced for more than a "short time". Once in tenure it normally won't be garbage collected util max heap is reached. So just because the heap memory usage isn't decreasing doesn't mean you have a leak.

The normal memory profile (heap used vs time) looks like a sawtooth - memory usage slowly increases until some threshold (typically max heap), and then decreases because of a GC, then slowly increases again.

Only thing I would add would be to try and connect a profiler to your application and see how the memory usage behaves. I've successfully profiled instances of Weblogic A.S. with both jProfiler and YourKit, although I haven't tried it with JBoss, should be fairly easy.

Can you replicate this behavior in a testing environment (since the profiler is kind of heavy on the performance, only do it in production environment if you're desperate)? If you do, you can see if the GC is only getting invoked when it reaches the XMX threshold (doesn't mean anything is wrong), and you can explicitly invoke a GC to see how it behaves. When you get a constant increase in memory even with GC calls, that might indicate a problem.

A decent profiler can tell you what objects are growing more rapidly in number and such, which can help you out tremendously if you do indeed have a "leak".