HttpWebRequest returns empty text

When I try to fetch th开发者_运维知识库e content of this URL http://www.yellowpages.com.au/qld/gatton/a-12000029-listing.html

using System.Net;

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(link);
request.AllowAutoRedirect = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
StreamReader objSR;
objSR = new StreamReader(resStream, System.Text.Encoding.GetEncoding("utf-8"));
string sResponse = objSR.ReadToEnd();

I don't get any response from the server. Please help me find out why this happens.

Thanks in advance!

It may well be looking at the user agent and refusing to serve content to a client that doesn't identify itself. Try setting the UserAgent property on your request object.

Looks to me like that site is checking the referrer url and may be serving up empty content if an invalid referrer is specified.

Try setting request.Referer = "http://www.google.com";. Experiment with the referrer to see if that changes the response. I'd also try the UserAgent property as Matthew suggested.

I had the same problem and the cause was that I previously had set the method to HEAD and in later revisions had the need to parse the body.