load test memory problem

We are testing our app running on IBM portal server in the Linux box but found that the "free" values of vmstat are decreasing steadily, even after the test. By looking at top, the "VIRT" values are also increasing steadily. By monitoring Java app heap usage, the initial heap size (1.5G) was never reached, and the usage was rising slowly and steadily (with minor rises/drops within the test period) from 6xxm to about 1g until the test ended. When the test just ended, it dropped by a large amount back to about 6XXm. My questions are: 1. Is the result normal and OK? 2. Is the app heap usage behaviour OK? 3. Is it normal that the "free" values of vmstat are decreasing steadily and "VIRT" values of top are increasing steadily without drop after the test?

Below are top and vmstat output:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14157 user01    17   0 2508m 1.2g  47m S 16.0 23.3  11:38.94 java               
14157 user01    17   0 2508m 1.2g  47m S 16.9 23.3  11:49.08 java               
14157 user01    17   0 2508m 1.2g  47m S 15.8 23.4  11:58.58 java               
14157 user01    17   0 2509m 1.2g  47m S 13.0 23.5  12:06.36 java               
14157 user01    17   0 2509m 1.2g  47m S 17.6 23.5  12:16.92 java               
14157 user01    17   0 2509m 1.2g  47m S 16.9 23.6  12:27.09 java               
14157 user01    17   0 2510m 1.2g  47m S 16.1 23.6  12:36.73 java               
14157 user01    17   0 2510m 1.2g  47m S 14.5 23.7  12:45.43 java               
...
14157 user01    17   0 2514m 1.2g  47m S 15.9 24.6  15:20.18 java               
14157 user01    17   0 2514m 1.2g  47m S 16.2 24.6  15:29.88 java               
14157 user01    17   0 2514m 1.2g  47m S 16.1 24.7  15:39.56 java               
14157 user01    17   0 2515m 1.2g  47m S 19.5 24.7  15:51.28 java               
14157 user01    17   0 2516m 1.2g  47m S 11.4 24.8  15:58.11 java               
14157 user01    17   0 2515m 1.2g  47m S 14.7 24.8  16:06.91 java               
14157 user01    17   0 2515m 1.2g  47m S 16.0 24.9  16:16.51 java               
14157 user01    17   0 2515m 1.2g  47m S 16.1 24.9  16:26.15 java               
14157 user01    17   0 2515m 1.2g  47m S 14.7 25.0  16:34.96 java               
14157 user01    17   0 2516m 1.2g  47m S 11.8 25.0  16:42.03 java               
...
14157 user01    17   0 2517m 1.3g  47m S 13.1 25.6  18:18.04 java               
14157 user01    17   0 2517m 1.3g  47m S 17.8 25.6  18:28.75 java               
14157 user01    17   0 2516m 1.3g  47m S 15.2 25.7  18:37.85 java               
14157 user01    17   0 2517m 1.3g  47m S 13.5 25.7  18:45.93 java               
14157 user01    17   0 2516m 1.3g  47m S 14.6 25.8  18:54.70 java               
14157 user01    17   0 2517m 1.3g  47m S 14.6 25.8  19:03.47 java               
14157 user01    17   0 2517m 1.3g  47m S 15.3 25.9  19:12.67 java               
14157 user01    17   0 2517m 1.3g  47m S 16.6 25.9  19:22.64 java               
14157 user01    17   0 2517m 1.3g  47m S 15.0 26.0  19:31.65 java               
14157 user01    17   0 2517m 1.3g  47m S 12.4 26.0  19:39.09 java               
...
14157 user01    17   0 2530m 1.4g  47m S  0.0 27.5  23:23.91 java               


procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0   2004 702352 571508 1928436    0    0     0    54  287  413  1  1 98  0  0
 0  0   2004 702368 571528 1928416    0    0     0    12  280  379  0  0 100  0  0
...
24  0   2004 673988 572504 1948000    0    0     0   440  760  751 16  6 78  0  0
 0  0   2004 671352 572540 1951048    0    0     0   477 1180  830 19  7 74  0  0
 0  0   2004 674756 572572 1946904    0    0     0   380  604  650 13  3 84  0  0
 1  0   2004 694208 572612 1928360    0    0     0   222  518  599  7  2 91  0  0
16  0   2004 692068 572640 1929360    0    0     0   539 1075  850 24  7 69  0  0
 0  0   2004 689036 572680 1931376    0    0     0   292  978  781 14  6 81  0  0
...
 0  0   2004 530432 579120 2007176    0    0     0   453  511  712 18  4 78  0  0
 0  0   2004 528440 579152 2008172    0    0     0   200  436  652 10  2 87  0  0
 0  0   2004 524352 579192 2010188    0    0     0   401  514  779 17  6 76  0  0
 0  0   2004 524964 578208 2012200    0    0     0   514  475  696 15  3 82  0  0
 0  0   2004 522484 578260 2013176    0    0     0   416  488  699 15  3 82  0  0
 2  0   2004 521264 578300 2015192    0    0     0   368  501  728 14  5 80  0  0
 0  0   2004 518400 578340 2016180    0    0     0   404  452  647 14  3 84  0  0
25  0   2004 517064 578368 2018208    0    0     0   414  497  752 15  3 82  0  0
...
 0  0   2004 499312 578820 2029064    0    0     0   351  459  660 13  3 84  0  0
 0  0   2004 496228 578872 2031068    0    0     0   260  473  701 15  5 80  0  0
 0  0   2004 501360 578912 2026916    0    0     0   500  398  622  9  3 88  0  0
 1  0   2004 499260 578948 2027908    0    0     0   262  436  638 13  2 85  0  0
 1  0   2004 497964 578984 2028900    0    0     0   276  452  628 15  3 82  0  0
 0  0开发者_开发问答   2004 497492 579024 2029888    0    0     0   200  384  548  7  2 91  0  0
 0  0   2004 496620 579044 2030896    0    0     0   172  393  586  9  2 89  0  0
...
 1  0   2004 357876 566000 2104592    0    0     0   374  510  736 18  6 76  0  0
23  0   2004 358544 566032 2105588    0    0     0   362  456  644 12  3 85  0  0
 0  0   2004 376332 566084 2087032    0    0     0   353  441  614 13  3 84  0  0
 0  0   2004 375888 566120 2088024    0    0     0   220  411  620 10  2 88  0  0
 0  0   2004 375280 566156 2087988    0    0     0   224  408  586  7  2 91  0  0
16  0   2004 373092 566188 2090012    0    0     0   233  494  723 12  3 85  0  0
 2  0   2004 369564 566236 2090992    0    0     0   455  475  714 14  5 80  1  0
...
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0   2004 235156 572776 2155384    0    0     0     8  282  396  0  0 100  0  0
 0  0   2004 235132 572796 2155364    0    0     0    24  291  435  0  0 100  0  0
 1  0   2004 234780 572828 2155332    0    0     0   101  292  474  1  5 94  0  0
 0  0   2004 234804 572844 2155316    0    0     0    45  288  451  0  1 99  0  0
 0  0   2004 234852 572856 2155304    0    0     0    12  283  409  0  0 100  0  0

Heap Usage:

The most likely cause that you are seeing is that the -Xms and -Xmx values differ in your JAVA_OPTS. What appears to be happening here is that the OS is allocating memory when needed by the JVM. It is never bad practice to allocate all of the heap you will need to the JVM on start-up. Set the 2 values to their upper limit. By setting the two values equal, the JVM never has to request memory from the OS and is free to do the work it needs to within its own memory space.

It is not uncommon to see the behavior you have seen, the JVM will continue to request more memory from the OS until the upper limits (-Xmx) are reached. If you are new to Sizing the heap, or other techniques around tuning the JVM, have a look at this guide.

On another note, top and vmstat will only show you so much of the picture into what is happening with the JVM's memory. What you are seeing is what the operating system is allocating to it. You will want to use other tools such as jmap and jvisualvm to see how the memory inside the JVM is responding. These tools will be a better bench mark for your application. What they will show you is the New and Old generations, Garbage collections and other stats which are really important.