how to do asynchronous http requests with epoll and python 3.1

UPDATE: after much laboring with Py3, including writing my own asynchronous webserver (following a presentation given by Dave Beazley), i finally dumped Python (and a huge stack of my code )-: in favor of CoffeeScript running on NodeJS. Check it out: GitHub (where you'll find like 95% of all interesting code these days), npm (package manager that couldn't be any user friendly; good riddance, easy_install, you never lived up to your name), an insanely huge repository of modules (with tons of new stuff being published virtually 24/7), a huge and vibrant community, out-of-the-box asynchronous HTTP and filehandling..., all that (thanks to V8) at one third the speed of light — what's not to like? read more propaganda: "The future of Scripting" (slide hosting courtesy SpreeWebdesign).

there is an interesting page http://scotdoyle.com/python-epoll-howto.html about how to do asnchronous / non-blocking / AIO http serving in python 3.

there is the tornado web server which does include a non-blocking http client. i have managed to port parts of the server to python 3.1, but the implement开发者_JS百科ation of the client requires pyCurl and seems to have problems (with one participant stating how ‘Libcurl is such a pain in the neck’, and looking at the incredibly ugly pyCurl page i doubt pyCurl will arrive in py3+ any time soon).

now that epoll is available in the standard library, it should be possible to do asynchronous http requests out of the box with python. i really do not want to use asyncore or whatnot; epoll has a reputation for being the ideal tool for the task, and it is part of the python distribution, so using anything but epoll for non-blocking http is highly counterintuitive (prove me wrong if you feel like it).

oh, and i feel threading is horrible. no threading. i use stackless.

people further interested in the topic of asynchronous http should not miss out on this talk by peter portante at PyCon2010; also of interest is the keynote, where speaker antonio rodriguez at one point emphasizes the importance of having up-to-date web technology libraries right in the standard library.

edit there may be many ways to achieve high throughputs with little resource consumption. however, it is not only me who thinks that abandoning threads (and other, partially outdated, methods); the people over at the google skipfish project seem to think similar: they say skipfish uses a

Multiplexing single-thread, fully asynchronous network I/O and data processing model that eliminates memory management, scheduling, and IPC inefficiencies present in some multi-threaded clients.

FWIW, I have a library at: http://github.com/mnot/nbhttp/

The client side is in production as part of redbot.org.

I haven't tried it in Python 3, but it may help as a basis for your work.

Cheers,

In this poll, months ago, Giampaolo Rodolá said

I have a patch for adding epoll() support which is almost ready (asyncore supports poll(), not epoll()), it's just a matter of writing tests.

I'm not sure where to find that patch, but it seems a simple enough task that it should be readily portable into asyncore (the standard python library approach to async servers and clients -- sure, way behind Twisted or Tornado, but that's what the stdlib has;-).

circuits has support for epoll and includes an async )#50 client.

http://circuitsframework.com/

Python 3 lacks libraries. You seem interested in libraries to help you do stuff, so why exactly are you using python 3 to solve your problem?