Why do the GC times increase steadily on a long running high volume Java app?

I have a high volume Java application that handles a consistent load of 50000msgs/sec. It is tuned for high throughput using the following settings:

-Xmx3g -Xms3g -XX:NewSize=2g -Xss128k -XX:SurvivorRatio=6 -XX:TargetSurvivorRatio=90 -XX:+UseParallelGC -XX:ParallelGCThreads=12 -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError

I am finding that the young GC times steadily creeps up fro开发者_StackOverflow中文版m 50ms when it starts to 200ms by the end of the day, although the frequency of GC runs remains the same.

If I try the same run using the ParNewGC collector, the GC times goes up at a much faster rate. Does anyone have any thoughts on this problem?

If you have a memory leak, or an in-memory cache that is gradually using more and more memory, these would have the effect of causing the GC to do more work tracing the reachable objects.

Other possibilities are:

• You have a non-heap memory leak, and that is resulting in increasing paging; i.e. copying of physical memory pages to disc and back.

• Some external process is consuming increasing amounts of memory, resulting in increasing paging.

• The nature of the objects generated in the young generation is changing over time so that more work is needed to trace the reachable objects. It may simply be that a larger proportion of them are surviving past the first collection.

It may be a combination of several factors:

• A real memory leak. ( Objects are still referenced by accident)
• More objects, or maybe a different distribution of the objects between the gc-pools.
• The heap is larger. Plot the gc-time and the heap size.
• Memory fragmentation. On OS level and/or the java heap, depending on the jvm.

Why don't you plot the number of objects, heap size and gc-time. (using a tool, or by quering MBeans) A plot should tell you the trend; increasing, decreasing or linear, and will help you to determine if there is a leak. If possible, also dump some statistics from your program, like the number of processed messages or active sessions etc.

Is it a real problem since your messages are not processed during gc?. Try the concurrent mark and sweep gc...

* -XX:+UseConcMarkSweepGC -XX:+UseParNewGC