Loading url with cyrillic symbols

I have to load some url with cyrillic symbols. My script should work with this:

http://wincode.org/%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5/

If I'll use this in browser it would replaced into normal symbols, but urllib code fails with 404 error. How to decode correctly this url?

When I'm using that url directly in code, like address = 'that address', it works perfect. But I used parsing page for gettin开发者_如何学Gog this url. I have a list of urls which contents cyrillic. Maybe they have uncorrect encoding? Here is more code:

requestData = urllib2.Request( %SOME_ADDRESS%, None,  {"User-Agent": user_agent})
requestHandler = pageHandler.open(requestData)

pageData = requestHandler.read().decode('utf-8')
soupHandler = BeautifulSoup(pageData)

topicLinks = []
for postBlock in soupHandler.findAll('a', href=re.compile('%SOME_REGEXP%')):
    topicLinks.append(postBlock['href'])

postAddress = choice(topicLinks)

postRequestData = urllib2.Request(postAddress, None,  {"User-Agent": user_agent})
postHandler = pageHandler.open(postRequestData)
postData = postHandler.read()

  File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

I have a list of urls which contents cyrillic.

OK, if it contains raw (not %-encoded) Cyrillic characters that's not like the example, and in fact it isn't a URL at all.

An address with non-ASCII characters in it is known as an IRI. IRIs shouldn't be used in an HTML link, but browsers tend to fix up these mistakes.

To convert an IRI to a URI which you can then open with urllib, you have to:

1. encode non-ASCII characters in the hostname part using Punycode (IDNA).

2. encode non-ASCII characters in rest of the IRI to UTF-8 bytes and URL-encode them (resulting in %D0%BF... like in the example URL).

an example implementation.

You can try to use the urllib.unquote method.

>>> import urllib
>>> string = urllib.unquote("http://wincode.org/%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5/")
>>> print string.decode("utf-8")
http://wincode.org/программирование/

the following code worked for me (modified code from Arseny above):

import urllib.parse
string='http://wincode.org/%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5/'
string = urllib.parse.unquote(string,encoding='utf-8') # http://wincode.org/программирование/