urllib2.urlopen throws 404 exception for urls that browser opens

The following url (and others like it) can be opened in a browser but causes urllib2.urlopen to throw a 404 exception: http://store.ovi.com/#/applications?categoryId=20&fragment=1&page=1

开发者_如何学JAVAgeturl() returns the same url (no redirect). The headers are copied and pasted from firebug. I tried passing in the headers as a dictionary to Request, but got the same result. wget opens the url in the console but not from the script.

the code:

source_url = 'http://store.ovi.com/#/applications?categoryId=20&fragment=1&page=2'

try:

    socket.setdefaulttimeout(10)
    hdrs = [('Host','store.ovi.com'),('User-Agent','Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US;rv:1.9.0.13) Gecko/2009073021 Firefox/3.0.13 AppEngine-Google;(+http://code.google.com/appengine)'),('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'),('Accept-Language','en-us,en;q=0.5'),('Accept-Encoding','gzip,deflate'),('Accept-Charset','ISO-8859-1,utf-8;q=0.7,*;q=0.7'),('Keep-Alive','115'),('Connection','keep-alive'),('Cookie','JNPRSESSID=4u4devdrt7eb6e0qem3gin47i2; s_cc=true; undefined_s=First%20Visit; s_nr=1282817443274; s_sq=%5B%5BB%5D%5D; view=Grid; menu=menuOpen; OVI_DEVICE=b5130'),('Cache-Control','max-age=0')]
ree = urllib2.Request(source_url)
    ree.addheaders = hdrs
    opener = urllib2.build_opener()
    htmlSource = opener.open(ree).read()

except urllib2.HTTPError, e:

    print e.code  
    print e.msg
    print e.headers

The error output:

404

Not Found

Date: Sat, 28 Aug 2010 00:36:57 GMT

Server: Apache/2.2.3 (Red Hat)

X-Powered-By: PHP/5.2.2

Expires: Thu, 19 Nov 1981 08:52:00 GMT

Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

Pragma: no-cache

Keep-Alive: timeout=7, max=333

Connection: Keep-Alive

Transfer-Encoding: chunked

Content-Type: text/html; charset=UTF-8

What, if anything, am I doing incorrectly? Is this a bug? And if so, is there a workaround? Thanks!

Given a URL like:

http://store.ovi.com/#/applications?categoryId=20&fragment=1&page=2

The bit that browsers fetch is just:

http://store.ovi.com/

Everything to the right of that is a ‘fragment identifier’, which is not passed to the server at all (evidently, if you try, it will get confused). Instead, the HTML returned for the / URL will include a load of JavaScript that reads the #... data at the client side and fills in the page content using a bunch of XMLHttpRequests.

Webapps implemented like this are a big old pain to scrape, because you can't just take the HTML content of the main page. Instead you have to either analyse the script to find out where it gets the actual data from, or you have to hook up a real browser in order to execute all the scripts and see what document objects you're left with. They're also typically bad for accessibility and SEO.

Luckily for you this site appears to be putting something in the fragment that's also a valid path. So it looks like you can get the dynamic page data from the URL:

http://store.ovi.com/applications?categoryId=20&fragment=1&page=1