Multiple (asynchronous) connections with urllib2 or other http library?
I have code like this.
for p in range(1,1000):
result = False
while result is False:
ret = urllib2.Request('http://server/?'+str开发者_高级运维(p))
try:
result = process(urllib2.urlopen(ret).read())
except (urllib2.HTTPError, urllib2.URLError):
pass
results.append(result)
I would like to make two or three request at the same time to accelerate this. Can I use urllib2 for this, and how? If not which other library should I use? Thanks.
You can use asynchronous IO to do this.
requests + gevent = grequests
GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily.
import grequests
urls = [
'http://www.heroku.com',
'http://tablib.org',
'http://httpbin.org',
'http://python-requests.org',
'http://kennethreitz.com'
]
rs = (grequests.get(u) for u in urls)
grequests.map(rs)
So, it's 2016
精彩评论