How can I scrape an ASP.NET site that does all interaction as postbacks?

Using Python, I built a scraper for an ASP.NET site (specifically a Jenzabar course searching portlet) that would create a new session, load the first search page, then simulate a search by posting back the required fields. However, something changed, and I can't figure out what, and now I get HTTP 500 responses to everything. There are no new fields in the browser's POST data that I can see.

I would ideally like to figure out how to fix my own scraper, but that is probably difficult to ask about on StackOverflow without including a ton of specific context, so I was wondering if there was a way to treat the page as a black box and just fire click events on the postback links I want, then get the HTML of the result.

I saw some answers on here about scraping with JavaScript, but they mostly seem to focus on waiting for javascript to load and开发者_Go百科 then returning a normalized representation of the page. I want to simulate the browser actually clicking on the links and following the same path to execute the request.

Without knowing any specifics, my hunch is that you are using a hardcoded session id and the web server's app domain recycled and created new encryption/decryption keys, rendering your hardcoded session id (which was encrypted by the old keys) useless.

You could try using Firebugs NET tab to monitor all requests, browse around manually and then diff the requests that you generate with ones that your screen scraper is generating.

If you are just trying to simulate load, you might want to check out something like selenium, which runs through a browser and handles postbacks like a browser does.