Parsing dependent requests in an http web response

I want to simulate the behaviour of the WebTestRequest class (in Visual Studio's Test Tools framework) where it can invoke dependent requests based on resources that are referred to in the response that is obtained from the original request.

For example, if I issue a web request and get the response by doing this:

string url = "http://www.mysite.com";
WebRequest request = WebRequest.Create(url);
using (WebResponse response = request.GetResponse())
{
    StreamReader reader = new StreamReader(response.GetResponseStream(开发者_如何转开发)); 
    string responseText = reader.ReadToEnd();
}

I would like to be able to parse responseText and see if there are any requests to other resources (like js/css files, images, etc.)

Is there an easy way of doing this? I hesitate to manually do this, as some of the resource requests may be set up programmatically and may not be obvious on a straightforward text parse.

Use a html/sgml parser library. I'm not familiar with Visual Studio, but there are frameworks for parsing html out there. Find one and look in the API for something related to finding elements.

I'm reasonably certain that WebTestRequest itself only does a "straightforward text parse" to determine the dependent requests, since it has no awareness of javascript. So if you were to implement such then your code would be accurately simulating the behaviour.

The following is a list of all the elements I could find in a cursory glance of the HMTL 4 spec that can refer to additional resources and thus would need to be parsed:

• <link href=
• <img src=
• <script src=
• <iframe src=
• <object data=
• <area href=

Not sure if it's exhaustive or not.

By the way, I'm curious as to what you ended up doing in the end.

EDIT:

some of the resource requests may be set up programmatically and may not be obvious on a straightforward text parse

It in fact becomes impossible at some point to determine the dependent requests from parsing an html response, and I'll give you an example: anything developed with Google Web Toolkit. In a recent GWT app that I tested, there was essentially no parsable html -- everything is run from javascript. Extracting obvious path names (when available) wasn't even useful because in reality conditional logic was selecting certain dependents and not others.