Sporadic behavior by the machines in stress

We are doing some Java stress runs (involving network IO). Initially things are all fine and the system responds very fast (avg latency in test 2ms). But hours later when I redo the same test I observe the performance goes down (20 - 60ms). It's the same Jar files, same JVM, and the same LAN over which 开发者_运维问答the stress is running. I am not understanding the reason for this behavior.

The LAN is 1GBPS and for the stress requirements I'm sure we are not using all of it.

So my questions:

1. Can it be because of some switches in the LANs?
2. Does the machine slow off after some time ( The machines are restarted .. say about 6months back well before the stress can start; They are RHEL5, XEON 64bit Quad core)
3. What is the general way to debug such an issues?

A few questions...

How much of the environment is under your control and are you putting any measures in place to ensure it's consistent for each run? i.e. are you sharing the network with other systems, is the machine you're using being used solely for your stress testing?

The way I'd look at this is to start gathering details on what your machine and code are up to. That means use perfmon (windows) sar (unix) to find out what the OS and hardware is doing and get a profiler attached to make sure your code is doing the same thing and help pin-point where the bottleneck is occuring from a code perspective.

Nothing terribly detailed but something I hope that will help get you started.

The general way is "measure everything". This, in particular might mean:

1. Ensure time on all servers is the same (use ntp or something similar);
2. Measure how long did it take to generate request (what if request generator has a bug?);
3. Measure when did request leave the client machine(s), or at least how long did it take to do i/o. Sometimes it is enough to know average time necessary for many requests.
4. Measure when did the request arrive.
5. Measure how long did it take to generate a response.
6. Measure how long did it take to send the response.

You can probably start from the 5th element, as this is (you believe) your critical chain. But it is best to log as much as you can - as according to what you've said yourself, it takes days to produce different results.

If you don't want to modify your code, look for cases where you can sniff data without intervening (e.g. define a servlet filter in your web.xml).