HTTP Request only works in browser

I have the following GET request that returns HTML开发者_如何学运维 for a login form, indicating that my authentication, i.e. the credentials are wrong. When I am authenticated in a browser session, and manually request the same URL, I get the expected empty XML document as a response. What am I missing?

    var getRequest = WebRequest.Create("http://frulo.com/v1/company/subscribers.xml") as HttpWebRequest;
    getRequest.Credentials = new NetworkCredential("user@company.net", "password");
    using (var response = getRequest.GetResponse() as HttpWebResponse)
    {
        var sr = new StreamReader(response.GetResponseStream());
        Response.Write(sr.ReadToEnd());
    }

You already know the answer: When you are authenticated in a browser session you get the correct response. That means that you are not authenticated when using the WebRequest.

The credentials that you provide are used for HTTP authentication but your web site uses most likely some sort of HTML forms-based authentication.

To solve the problem you will have to use the same authentication mechanism that the web application does. This might be cookie based or a session id might be transmitted as a POST or GET parameter along with each request. Without knowing further details about the web site it's difficult to provide more help though.

The following question is related and most likely of help for you:

C# Login to Website via program

They may be blocking your program's user agent to prevent this sort of scraping.

Using the Web Request Proxy property might help in some circumstances. I have inserted a line of code with a comment by MSDN.

   var getRequest = WebRequest.Create("http://frulo.com/v1/company/subscribers.xml") as HttpWebRequest;

    //MSDN states: Returns a proxy configured with the Internet Explorer settings of the currently impersonated user.
    getRequest.Proxy = WebRequest.GetSystemWebProxy();

    getRequest.Credentials = new NetworkCredential("user@company.net", "password");
    using (var response = getRequest.GetResponse() as HttpWebResponse) {
        var sr = new StreamReader(response.GetResponseStream());
        HttpContext.Current.Response.Write(sr.ReadToEnd());
    }

.