Why are the load averages for python threads varying for different linux cloud provider setups although the spec is the same?

We have a Python application that polls directories using threads and inotify watchers. We have always run this application in a cloud server provided by Voxel (http://www.voxel.net). We are currently testing a different cloud server provider StormOnDemand (http://stormondemand.com) and when we ran our application, our load averages were a lot higher then they were when they were running on the Voxel cloud server despite the specs being about the same (Refer below for more details on setup). We have also ensured then when testing the server was not handling any other loads.

I have written a simple test application (test_threads.py - attached, or refer to http://pastebin.com/xGQU7JD0) that simulates the issues we are seeing by starting up threads that loops, sleeping for a user defined time on each loop. It takes 2 parameters, the amount of threads to start and the interval period.

When I run, "python test_threads.py 50 0.1" for about 10 minutes

Load average results:

StormOnDemand: $ uptime 18:46:22 up 7:29, 6 users, load average: 4.43, 4.16, 2.93

Voxel $ uptime 18:48:14 up 9 days, 15:09, 9 users, load average: 0.51, 0.47, 0.43

Th开发者_Python百科e load average on the StormOnDemand server is a lot higher.

Python version:

StormOnDemand - 2.6.5

Voxel - 2.6.5

Server spec:

StormOnDemand - 8 x Intel(R) Xeon(R) CPU E5506 @ 2.13GHz; 16GB RAM; 230GB HDD (Storm Bare Metal servers)

Voxel - 7 x Intel(R) Xeon(R) CPU L5640 @ 2.27GHz; 14GB RAM; 200GB HDD (VoxCloud servers)

OS:

StormOnDemand - Ubuntu 10.04 - 2.6.36-rc8101910 #1 SMP Tue Oct 19 19:18:34 UTC 2010 x86_64 GNU/Linux

Voxel - Ubuntu 10.04 - 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 x86_64 GNU/Linux

Virtualisation method:

StormOnDemand - Not 100% sure, but I think they use Xen

Voxel - Not sure, but the image that we are using looks to us like a stock standard Ubuntu 10.04 server

Any suggestion on why the load would be a lot higher or how I could debug this further is greatly appreciated.

In general, trying to evaluate the multi-threading capabilities of a machine in a single Python interpreter is not a very good idea becasue of the GIL: at any time, only one of the threads of the interpreter is ready to run.

Now load average on Linux is the exponential average of the number of runnable threads with different damping factors. From this it follows that on the StormOnDemand system there must have been other processes running at the same time (4 runnable - 1 Python = 3 others). (Note that even the 4 runnable processes do not represent a high load on an 8-way system.)

To improve your measurements you may want to read out from /proc/self/stat the number of clock ticks the process is scheduled just before it exits. This will allow the comparison of the CPU speeds of the two hosts.

The higher load average may also be related to the hypervisor (XEN) and to running in a virtualized environment. The higher average does not necessarily indicate a performance problem: whether it is a problem depends on what your critical performance metric is.

E.g. to find out the maximum throughput (available capacity) of the hosts you have to try to saturate them with a representative workload. If the responsivity is a concern (i.e. how fast it can react to a stimulus), you need to measure the response time (possibly close to your end-users, so that network latency is included).

Perhaps the most important criterion of the comparison is however a contractual one, namely the Service Level Agreement your provider signs with you. This should contain the technical description of how the service levels are measured and center on the metric critical to your application.