Java 1.5 gc tuning

We have a large application with allocated heap of 2 GB min & 8 GB max. During load test we are finding some really long pause time the GC cycles are as large as 16+ secs. Initially we were using "-XX:+UseParNewGC" but switching to UseParallelGC gave us much needed performance boost, but we do have issue of longer pause time under load.

We tried few options such as increasing young generation but nothing seems to help any idea what else can be tried ? We have liberty to increase heap size if required but I am wondering that might worsen gc pause. If nothing can be done I am thinking of using cluster app servers with 5 GB heap instead of 1 larger heap. snap shot of current gc logs is attached

J  
J Thu Jun 23 12:40:56 2011
J  [GCJ  
J Thu Jun 23 12:40:57 2011
 [PSYoungGen: 2130792开发者_开发技巧K->475247K(2084160K)] 7198716K->5543171K(7676608K), 1.3280110 secs] [Times: user=0.00 sys=1.88, real=1.33 secs] 
J  
J Thu Jun 23 12:41:00 2011
J  [GCJ  
J Thu Jun 23 12:41:01 2011
 [PSYoungGen: 1966319K->417801K(1908928K)] 7034243K->5546416K(7501376K), 0.7025950 secs] [Times: user=0.01 sys=1.89, real=0.71 secs] 
J  
J Thu Jun 23 12:41:12 2011
J  [GCJ  
J Thu Jun 23 12:41:13 2011
 [PSYoungGen: 1908873K->269608K(2155520K)] 7037488K->5523748K(7747968K), 1.3117340 secs] [Times: user=0.01 sys=1.44, real=1.31 secs] 
J  
J Thu Jun 23 12:41:33 2011
J  [GC [PSYoungGen: 1747432K->138147K(1616000K)] 7001572K->5593865K(7208448K), 0.4949960 secs] [Times: user=0.01 sys=1.40, real=0.50 secs] 
J  [Full GCJ  
J Thu Jun 23 12:41:50 2011
 [PSYoungGen: 138147K->0K(1616000K)] [PSOldGen: 5455718K->3456287K(5592448K)] 5593865K->3456287K(7208448K) [PSPermGen: 256273K->256273K(524288K)], 17.0259440 secs] [Times: user=0.00 sys=16.88, real=17.02 secs] 
J  
J Thu Jun 23 12:42:09 2011
J  [GC [PSYoungGen: 1477824K->85118K(2110848K)] 4934111K->3541406K(7703296K), 0.1437050 secs] [Times: user=0.00 sys=0.30, real=0.14 secs] 
J  
J Thu Jun 23 12:42:20 2011
J  [GC [PSYoungGen: 1573438K->71812K(2100352K)] 5029726K->3600767K(7692800K), 0.2477960 secs] [Times: user=0.00 sys=0.65, real=0.25 secs] 

Looking at your log, your largest collection (the 17 second one) is collecting the old (tenured) generation. Using the concurrent garbage collector on the Sun JVM should help (+XX:UseConcMarkSweepGC), as this will do the collection (mostly) concurrently, reducing the pause time).

Your young generation pauses are reasonably large too, for the volume of data that's being collected. What spec of machine are you running on? These pauses aren't very frequent too, so if your goal is lower pause times, try reducing the size of the young generation (-XX:NewRatio), which should lead to shorter, more frequent pauses.

You should also be sure that there's no swapping happening on your machine. You don't say what OS you're runnning, but on Linux run:

vmstat 5

and check the "si" and "so" columns while these large GCs are taking place. If they are non-zero, either reduce the memory usage on the machine, or tweak the "swappiness" tunable.

Consider making your -Xms and -Xmx values the same. if you are willing to give the heap a max size of 8GB, then make that the min as well. This way the JVM doesn't have to worry about allocating more memory.

How frequent are the full GC's? if they are occurring frequently, then you may need to profile the application rather then beating your head against tuning gc parameters that will not matter in the end. Find out what is allocating all the memory for you and see if there is a way to reduce it.

On JDK 1.5 and JDK 1.6 you will see increasing GC pauses as you pass 2 to 4 GB Heap memory. Clustering applications that need larger data heaps is one option. The other I know of is considering alternate JVM's. If you are at 8 GB heap you may see improvements on JDK 1.7. There are some optimizations. Additionally, Sun / Oracle has several other JVM's to consider.

If you have not read this article, it is a very strong one and for Java 5.0 / JDK 1.5. It may help. Tuning Java 5.0. I would go into more detail, but the comments above do cover some of the elements.

Alternatively there is one other JVM on the market that is from a private company called Azul Systems. They have some open source tools to check if their C4 collector can help. They have a JVM platform that runs in parallel while your application is running. If their JHiccup tool says you have GC issues then they can provide an enterprise JVM. Cost is not free, so if you are looking at small deployments (one server) and problems that are less than $10,000 in value then I would (1) consider alternate free JVM and (2) try a different JVM if you can run on it (e.g. JDK 1.6 / 1.7). If none of these are an option try clustering but do so with Heaps below 4GB.

The only point I make here is that there "ARE" options on choice of JVM. JRocket does help some apps.