problem with VB.net WebRequest return page with no results

I am trying to get the html response of a Kayak.com query using a WebRequest object, however more often that not the site returns no results, i just get an empty Kayak page, i have to try the request around 5 times before i get a result back...What can be the cause of this? My code is given below...Thanks for the input! Fernando

Dim WReq As We开发者_运维技巧bRequest = WebRequest.Create("http://www.kayak.com/flights/MIA-LAX/2011-01-29/2011-02-11")
WReq.Timeout = 100000

Dim wResp As WebResponse = WReq.GetResponse()
Dim r As StreamReader = New StreamReader(wResp.GetResponseStream(), Encoding.ASCII)

Is this all of your code? Are you doing anything further with the StreamReader? If not, that's your problem. Once you've got the stream you need to read it into something that you can work with:

    Dim WReq As WebRequest = WebRequest.Create("http://www.kayak.com/flights/MIA-LAX/2011-01-29/2011-02-11")
    WReq.Timeout = 100000

    Dim wResp As WebResponse = WReq.GetResponse()
    Dim T As String
    Using r As StreamReader = New StreamReader(wResp.GetResponseStream(), Encoding.ASCII)
        T = r.ReadToEnd()
    End Using
    Trace.WriteLine(T)

If you've done this but just didn't post your code then you might be having a problem with the HTTP UserAgent header that's being sent. You might want to adjust that by doing something like:

DirectCast(WReq, HttpWebRequest).UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 ( .NET CLR 3.5.30729; .NET4.0E)"

A completely unrelated problem with your code is that you're assuming that the content is ASCII which will throw you off if there are any characters above the 127 barrier. You should inspect the headers to determine what encoding to use but since you've got a known site you can just pre-determine this and know that KAYAK uses UTF-8. So your StreamReader should be set to that instead.