Basic fetching of a URL's HTML body with Python 3.x
I'm a Python newbie. I have been a little confused by the differences between the old urllib and urllib2 in Python 2.x and the new urllib in Python 3, and among other things I'm not sure when data needs to be encoded before being sent to urlopen.
I have been trying to fetch the html body of a url, using a POST so that I can send parameters. The webpage displays sunshine data for a country over a particular hour of a given day. I have tried without encoding/decoding and the printout is a string of bytes with b at the beginning. The code I then tried was
import urllib.request, urllib.parse, urllib.error
def scrape(someurl):
try:
values = {'LANG': 'en',
'开发者_JAVA技巧DATE' : '1303160400',
'CONT' : 'euro',
'LAND' : 'UK',
'KEY' : 'UK',
'SORT': '2',
'INT' : '06',
'TYPE' : 'sonnestd',
'ART' : 'karte',
'RUBRIK' : 'akt',
'R': '310',
'CEL': 'C'}
data = urllib.parse.urlencode(values)
data = data.encode("utf-8")
response = urllib.request.urlopen(someurl, data)
html = response.read().decode("utf-8")
print(html)
except urllib.error.HTTPError as e:
print(e.code)
print(e.read())
myscrape = scrape("http://www.weatheronline.co.uk/weather/maps/current")
The error is
Traceback (most recent call last):
File "/Users/Me/Desktop/weather.py", line 57, in <module>
myscrape = scrape("http://www.weatheronline.co.uk/weather/maps/current")
File "/Users/Me/Desktop/weather.py", line 37, in scrape
html = response.read().decode("utf-8")
UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 10: invalid start byte
Without encoding/decoding I get a suspiciously short string of bytes anyway, so I wonder whether the request is failing in some other way
b'GIF89a\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\x00\x00\x00!\xf9\x04\x01\x00\x00\x00\x00,\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02D\x01\x00;'
The GIF89a
means that the server is sending you an image.
Also, you shouldn't be brute-force decoding with UTF-8 anyway; you should look at the response headers to figure out which encoding to use.
精彩评论