Is there a production-safe way to measure time spent in Production w/Python?

I want to be able to instrument Python applications so that I know:

• Page generation time.
• Percentage of time spent in external requests (mysql, api calls).
• Number of mysql queries, what the MySQL queries were.

I want this data from production (not offline profiling) - because the time spent in various开发者_开发百科 places will be different under load.

In PHP I can do this with XHProf or instrumentation-for-php. In Ruby on Rails/.NET/Java, I can do this with New Relic.

Is there such a package recommended for Python or django?

Yes, it's perfectly possible. E.g. use some magic switch in URL, like "?profile-me" which triggers profiling in Django middleware.

There are a number of snippets on the Internet, like this one: http://djangosnippets.org/snippets/70/ or modules like this one: http://code.google.com/p/django-profiling/ - but I haven't used any of them so I cannot recommend anything.

Anyway, the approach they take is similar to what I do - i.e. use Python Hotshot profiler module in a middleware that wraps your view. For the MySQL part, you can just use connection.queries form Django.

The nice thing about Hotshot is that its output can be browsed using Kcachegrind like here: http://www.rkblog.rk.edu.pl/w/p/django-profiling-hotshot-and-kcachegrind/

New Relic now had a package for Python, including Django through mod_wsgi.

https://support.newrelic.com/help/kb/python

django-prometheus is a good choice for handling production workloads, especially in a container environment like Kubernetes. Out of the box, it has middleware for tracking request latencies and counts (by view method), as well as Database and cache access times. It wouldn't be a good solution for tracking which queries are actually executing, but that's where a logging solution like ELK would come into play. If it helps, I've written a post which walks through how to add custom metrics to a Django application.