Benchmarking, Profiling on Virtual Machines

On many different sources you can read about time keeping issues in virtual machines. As every benchmark relies on time keeping i am not sure how to interpret e.g. apache benchmark or xdebug profiler results on vmware and how credible they are.

http://communities.vmware.com/docs/DOC-5581

VMware suggests to build a special vm for performance testing. There are many tips in addition which seems to be workaround solutions (install VMware Tools, special configuration and so on).

Especially when it comes to disk i/o performance issues i´m very concerned about how accurate the results are. Rasmus Lerdorf talked about it at drupalcon, saying that some of his performance improvements regarding stat calls aren´t visible in his VMware benchmarks because VMware has special disk caching you can´t compare to physical d开发者_JAVA技巧isks:

http://www.nowpublic.com/tech-biz/rasmus-lerdorf-simple-hard-drupalcon-2008-key-note

After all this I´m not sure if VMware is capable for any benchmarking at all.

A lot depends on what you are trying to do.

1. Just keep measurements to look for regressions
2. Find time-drains that you can fix to improve performance

Many people think these are the same, but I think that comes from limited experience actually doing (2).

The way I do (2) is not measuring, but asking, at a few random points in time, what is happening and why. For that, it doesn't matter very much if it's on a virtual machine or not. If it's doing something sneaky that takes a large fraction of time, you will see that whether or not it's on a virtual machine. The exact fraction doesn't matter. Exposing the sneaky activity is what matters.

Here's an example in python, but you can do it in any language.

Well it may be true about disk caching and slow performance without vmware-tools.
But I've found Vmware to be very helpful in development as I have a Linux Virtual Machine as my dev server, while I code in Windows.
And I can easily benchmark and find potential bottlenecks in my code.

It may not be perfect and might miss some things, but still it is much easier and cheaper than setting up a dedicated Linux box.

Update
ApacheBench is a good tool for testing scalability and raw RequestPerSeconds metric.

But for finding Bottlenecks in my PHP scripts and testing out code optimization effects, I always use XDebug along with either Webgrind or WinCacheGrind to view profiler output.

If you have a linux box you can install XDebug using pecl install xdebug or for windows you can find prebuilt binaries on the xdebug site.

I use the following config for Xdebug

[xdebug]
xdebug.profiler_enable = 0
xdebug.profiler_enable_trigger = 1
xdebug.profiler_output_dir = "/opt/lampp/htdocs/profiles"
xdebug.profiler_output_name = "cachegrind.out.%s.%t"

with profiler_enable_trigger I can selectively profile if needed by passing XDEBUG_PROFILE as query string parameter.
Also I have set up webgrind so I can directly view the output graphiclly in the browser itself.