WebCrawling User-authenticated websites

Is 开发者_开发技巧it possible to crawl user-authenticated websites using c#?

Yes, it is possible assuming you have the credentials. Depending on the authentication mechanism used by the site different methods exist. WebClient could be used to send HTTP requests to the remote sites.

To crawl user-authenticated websites, your crawler should be able to handle cookies and POST-requests.

The question is: do you want your crawler to sign up, validate itself by email and then sign in to crawl all contents?