Python 2 vs. Python 3 - urllib formats
I'm getting really tired of trying to figure out why this code works in Python 2 and not in Python 3. I'm just trying to grab a page of json and then parse it. Here's the code in Python 2:
import urllib, json
response = urllib.urlopen("http://reddit.com/.json")
content = response.read()
data = json.loads(content)
I thought the equivalent code in Python 3 would be this:
import urlli开发者_开发知识库b.request, json
response = urllib.request.urlopen("http://reddit.com/.json")
content = response.read()
data = json.loads(content)
But it blows up in my face, because the data returned by read() is a "bytes" type. However, I cannot for the life of me get it to convert to something that json will be able to parse. I know from the headers that reddit is trying to send utf-8 back to me, but I can't seem to get the bytes to decode into utf-8:
import urllib.request, json
response = urllib.request.urlopen("http://reddit.com/.json")
content = response.read()
data = json.loads(content.decode("utf8"))
What am I doing wrong?
Edit: the problem is that I cannot get the data into a usable state; even though json loads the data, part of it is undisplayable, and I want to be able to print the data to the screen.
Second edit: The problem has more to do with print than parsing, it seems. Alex's answer provides a way for the script to work in Python 3, by setting the IO to utf8. But a question still remains: why is it that the code worked in Python 2, but not Python 3?
The code you post is presumably due to wrong cut-and-paste operations because it's clearly wrong in both versions (f.read()
fails because there's no f
barename defined).
In Py3, ur = response.decode('utf8')
works perfectly well for me, as does the following json.loads(ur)
. Maybe the wrong copys-and-pastes affected your 2-to-3 conversion attempts.
Depends of your python version you have to choose the correct library.
for python 3.5
import urllib.request
data = urllib.request.urlopen(url).read().decode('utf8')
for python 2.7
import urllib
url = serviceurl + urllib.urlencode({'sensor':'false', 'address': address})
uh = urllib.urlopen(url)
Please see that answer in another Unicode related question.
Now: the Python 3 str
(which was the Python 2 unicode
) type is an idealised object, in the sense that it deals with “characters”, not “bytes”. These characters, in order to be used for/from disk/network data, need to be encoded-into/decoded-from bytes by a “conversion table”, a.k.a encoding a.k.a codepage. Because of operating system variety, Python historically avoided to guess what that encoding should be; this has been changing over the years, but still the principle of “In the face of ambiguity, refuse the temptation to guess.” applies.
Thankfully, a web server makes your work easier. Your response
above should give you all extra information needed:
>>> response.headers['content-type']
'application/json; charset=UTF-8'
So, every time you issue a request to a web server, check the Content-Type header for a charset value, and decode the request's data into Unicode (Python 3: bytes.decode(charset)
→ str
) by using that charset.
Here is an approach that is compatible across both versions - it works by first converting bytes data to string, and then loading the string.
import json
try:
from urllib.request import Request, urlopen #python3+
except ImportError:
from urllib2 import Request, urlopen #python2
url = 'https://jsonfeed.org/feed.json'
request = Request(url)
response_json_string = urlopen(request).read().decode('utf8')
response_json_object = json.loads(response_json_string)
精彩评论