Python Backend Design Patterns

I am now working on a big backend system for a real-time and history tracking web service.

I am highly experienced in Python and intend to use it with sqlalchemy (MySQL) to develop the backe开发者_C百科nd.

I don't have any major experience developing robust and sustainable backend systems and I was wondering if you guys could point me out to some documentation / books about backend design patterns? I basically need to feed data to a database by querying different services (over HTML / SOAP / JSON) at realtime, and to keep history of that data.

Thanks!

Can you define "backend" more precisely? Normally, in web dev, I follow a MVC'ish structure where my "front-end", html/css/js and code dealing with displaying either, is loosly coupled with my "backend" model (business objects and data persistence; i.e. database).

I like Django's Model/View/Template approach:

http://docs.djangoproject.com/en/dev/faq/general/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names

But, you haven't really defined what you mean by "backend" so its hard to give advice on design patterns. You said you are experienced in Python, have you ever developed a database driven web application before?

update

Based on your comment, I won't be able to help much as I don't have much experience doing "backends" like that. However, seeing as how you are pulling in resources from the web, your latency/throughput is going to be pretty high. So, in order to increase overall effectiveness, you are going to want to have something that can run multiple threads or processes with pretty high concurrency. I suggest you check out the answers on this thread (and search for similar ones):

Concurrent downloads - Python

Specifically, I found the example for the recursive web server and the example following it to probably be a very good start on your solution:

http://eventlet.net/doc/examples.html#recursive-web-crawler

As for taking that idea and then turning it into a robust/continuous process, that's going to depend a lot on your platform and how well you do error handling. Basically:

• run it in a loop and make sure you handle any error that can possibly be thrown
• have some kind process monitoring your worker process to kill/restart it if it hangs or dies
• make sure you have a monitoring solution to notify you if it stops working (nagios, etc.)

One of the best ways to keep things "robust" is to make them as simple (not simplistic) as possible. If all you are doing is pulling in info from the web, parsing it in some way, and then storing that info in a DB, then try to keep the process that simple. Don't add unnecessarily complexity in an effort to make it more robust. If you end up with a 200 line script that does what you want, great!

Use Apache, Django and Piston.

Use REST as the protocol.

Write as little code as possible.

Django models, forms, and admin interface.

Piston wrapppers for your resources.