Problem in downloading twitter search pages

I am trying a small program to download webpages and collect some data. The program works fine for most webpages. However, there is problem in downloading Twitter search results. The download program seems to get only twitter homepage. I am using WinInet for downloading.

N开发者_如何学编程oticed Twitter uses "#!" in its url. (www.twitter.com/#!/...). Is it something to do with this?

Appreciate any suggestion on how to handle this.

Thanks

If you want to collect some data from Twitter you should use their API.

Just remove the hashbang and it will be fine. However, using the API is more appropriate.

You are probably falling victim to the cache, especially with # in the URL. Try INTERNET_FLAG_RELOAD | INTERNET_FLAG_NO_CACHE_WRITE (ignore cache and reload from network | do not bother writing response to cache) for dwFlags in HttpOpenRequest or InternetOpenURL. You could also try URLEncoding the #.