开发者

Does Wikipedia allow URL fetching via Google App Engine?

I am writing a Python web app and in it I plan to leverage Wikipedia. When trying out some URL Fetching code I was able to fetch both Google and Facebook (via Google App Engine services), but when I attempted 开发者_开发技巧to fetch wikipedia.org, I received an exception. Can anyone confirm that Wikipedia does not accept these types of page requests? How can Wikipedia distinguish between me and a user?

Code snippet (it's Python!):

    import os
import urllib2
from google.appengine.ext.webapp import template


class MainHandler(webapp.RequestHandler):
    def get(self):
        url = "http://wikipedia.org"
        try:
          result = urllib2.urlopen(url)
        except urllib2.URLError, e:
          result = 'ahh the sky is falling'
        template_values= {
            'test':result,
        }
        path = os.path.join(os.path.dirname(__file__), 'index.html')
        self.response.out.write(template.render(path, template_values))


urllib2 default user-agent is banned from wikipedia and it results in a 403 HTTP response.
You should modify your application user-agent with something like this:

#Option 1
import urllib2
opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'MyUserAgent')]
res= opener.open('http://whatsmyuseragent.com/')
page = res.read()

#Option 2
import urllib2
req = urllib2.Request('http://whatsmyuseragent.com/')
req.add_header('User-agent', 'MyUserAgent')
urllib2.urlopen(req)

#Option 3
req = urllib2.Request("http://whatsmyuseragent.com/", 
                       headers={"User-agent": "MyUserAgent"})
urllib2.urlopen(req)

Bonus link:
High level Wikipedia Python Clients http://www.mediawiki.org/wiki/API:Client_code#Python


You can set your user-agent to any string you wish; it will be modified by App Engine to append the string AppEngine-Google; (+http://code.google.com/appengine; appid: yourapp). In urllib2, you can set the user-agent header like this:

req = urllib2.Request("http://en.wikipedia.org/", headers={"User-Agent": "Foo"})
response = urllib2.urlopen(req)
0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜