exact way to measure performance on individual methods from inside Java?

I'd like to measure the execution time of a set of unit tests to be able to automatically monitor and compare performance when changes are made. Are there any suitable performance counters that can be reached easily from Java itself? Ie, something like:

count1 = <call performance counter func>
executeUnitTest();
count2 = <call performance counter func>
testPerformance = count2 - count1;

with these additional requirements:

• gives the same answer independent of system load, even if the test runner is run in a virtualized guest OS (this seems to disqualify nanoTime() and friends, or even current thread's cpu time, due to varying load on the host machine)
• gives the same answer independent of what hardware it is run on (counting number of executed bytecode instructions would be exce开发者_如何学编程llent!)
• ideally, can separate GC execution count from actual method execution (in case GC is done in-thread)

The only way to do this is to instrument your code to add a platform independent counter. You can add into your code where ever you have unit(s) of work.

Use AtomicLong/AtomicInteger if you want it to be thread safe.

 public static int counter;


 // instrumented into the code.
 counter++;

 // big piece of work
 counter+=5;

You can then compare how much the counter increases by. This will be independent of the speed of the CPU, GCs, even hibernating the application in the middle might not matter.

The approach I would take is to compare many results with another similar reference implementation I wouldn't expect to change. You can expect a certain ratio and this ration will only vary by a small amount regardless of the platform you are running on.

There are a couple of approaches you could consider.

'Instrumenting' happens at compile- or load-time, and alters the class files so that they keep counters of what is going on. Typically, this is used for doing things like measuring unit tests' code coverage, but you could probably modify a code coverage tool to just keep execution counts. From memory, think that Emma, the code coverage tool, is a good open-source place to start. Since GC doesn't get instrumented, that won't get counted. There may well be something that does it out-of-the-box.

Profiling tools probably can give you at least some of the data you are after. There are heaps available, including the standard 'java' executable being able to generate a lot of information that you might be able to use.

There's also quite sophisticated profiling hooks available in the VM, the API is reachable via the standard JVMTI api, which is as of IIRC 1.5 a part of standard API.

You won't ever get an exact measure (short of hypothetical approaches like running a whole machine simulator).

All of the practical approaches suffer from the problem that the measurement process distorts the behavior of the application being measured. For instance inserting code to capture the system clock time before and after is liable to hamper the JIT compiler's ability to optimize and alter the memory access patterns. Profiling tools that work by modifying the code suffer from this problem too. Other tools that work by sampling the code pointer suffer from the problem that they require a lot of samples to be statistically significant.

In short, while you can get useful results from ad-hoc measurement and profiling, the results need to be taken with a grain of salt. They can be misleading, and they can easily be misinterpreted.

I'd recommend that you base your final decisions about whether your code tuning helps on large-scale (system level) benchmarking using realistic workloads.