开发者

Multiple urllib2 connections

I want to download multiple images at the same time. For that I'm using threads, each one downloading an image, using urllib2 module. My problem is that even if threads starts (almost) simultaneously, the images are downloaded one by one, like in a single-threaded environment.

Here is the threaded function:

def updateIcon(self, iter, imageurl):
    req = urllib2.Request('http://site.com/' + imageurl)
    response = urllib2.urlopen(req)
    imgdata = response.read()
    gobject.idle_add(self.setIcon, iter, imgdata)

Debugging my code I found that downloads seems to get stuck at "response = urllib2.urlopen(req)" line. What's the problem? It's because the threading module or urllib2? How I can fi开发者_运维知识库x that?

Thank you in advance


Consider using urllib3. It supports connection pooling and multiple concurrent requests via processes (not threads). It should solve this problem. Be careful to garbage collect connection pools if you contact many different sites, since each site gets its own pool.


In my experience, multithreads of CPython seems to make better performance than those of sigle thread. Because CPython has thread implementation based on kernel thread. But the difference is little, because of GIL(Global Interpreter Lock). Substitute multiprocessing for multithreading. It's easy. Both have similar interface.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜