Using HTTP proxy using Socket
I am required to download web content including full headers like
HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/html; charset=utf-8
Expires: Sat, 30 Jul 2011 06:19:13 GMT
P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"
Date: Sat, 30 Jul 2011 06:20:13 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Connection: Transfer-Encoding
Set-Cookie: _SS=SID=0B3A2FD5AA7943BC92252BB73BD7C9CA; domain=.bing.com; path=/
Set-Cookie: MUID=CE6F495249204D82A8F620B7317FC59E; expires=Mon, 29-Jul-2013 06:20:13 GMT; domain=.bing.com; path=/
Set-Cookie: OrigMUID=CE6F495249204D82A8F620B7317FC59E%2c95e9e1eafdef40d6a24497335843fac6; expires=Mon, 29-Jul-2013 06:20:13 GMT; domain=.bing.com; path=/
Set-Cookie: OVR=flt=0&flt2=0&flt3=0&flt4=0&flt5=0&flt6=0&flt7=0&flt8=0&flt9=0&flt10=0&flt11=0&ramp1=snrport4-release&release=or3&preallocation=0&R=1; domain=.bing.com; path=/
Set-Cookie: SRCHD=D=1881020&MS=1881020&AF=QBLH; expires=Mon, 29-Jul-2013 06:20:13 GMT; domain=.bing.com; path=/
Set-Cookie: SRCHUID=V=2&GUID=A2EAC1B8990D46619C897016C94B5C4B; expires=Mon, 29-Jul-2013 06:20:13 GMT; path=/
Set-Cookie: SRCHUSR=AUTOREDIR=0&GEOVAR=&DOB=20110730; expires=Mon, 29-Jul-2013 06:20:13 GMT; domain=.bing.com; path=/
000037E4
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml" xmlns:Web="http://schemas.live.com/Web/"><head><meta content="text/html; charset=utf-8" http-equiv="content-type" /><script type="text/javascript">//<![CDATA[
Since content with full headers is not available in WebClient, HttpWebRequest i am using Socket for the same, here is the code.
using (Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.IP))
{
IPHostEntry entry = Dns.GetHostEntry(fullUrlAddress);
socket.ReceiveTimeout = 3000;
socket.Connect(entry.AddressList[0], 80);
string request = string.Empty;
string build_request = string.Empty;
if (cookieJar.Count != 0)
{
request = "GET {0} HTTP/1.1\r\nHost: {1}\r\nUser-Agent: Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: keep开发者_如何学编程-alive\r\nReferer: {0}\r\nCookie: {2}\r\n\r\n";
build_request = string.Format(request, requestedUri.AbsoluteUri, requestedUri.Host, GetCookies(requestedUri));
}
else
{
request = "GET {0} HTTP/1.1\r\nHost: {1}\r\nUser-Agent: Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: keep-alive\r\nReferer: {0}\r\nCookie: {2}\r\n\r\n";
build_request = string.Format(request, requestedUri.AbsoluteUri, requestedUri.Host, "PREF=ID=19495678a6a3dd6e:U=c5ce8e4e3f61da69:FF=0:TM=1311310634:LM=1311310636:S=gbV7hD2dPfycsf8Q; NID=49=dN3QceFFBFxwsCXM43HCRJF_oxoBpUHuUWt2tpoofEDFcRhj7TWWV4EFQNuVYP1GhyBAsQr3oOeohsJp31x8kb_iXiGcQFh1a3IFsPTNKjzJv_NgSK8ssG956PJO7jH-");
}
byte[] data = Encoding.UTF8.GetBytes(build_request);
socket.Send(data, data.Length, 0);
int bytes = 0;
byte[] bytesReceived = new byte[10240];
string currentBatch = string.Empty;
try
{
do
{
bytes = socket.Receive(bytesReceived);
currentBatch = Encoding.ASCII.GetString(bytesReceived, 0, bytes);
responseString.Append(currentBatch);
}
while (bytes > 0);
}
catch (Exception)
{
}
socket.Close();
}
It is working perfectly, but i don't know how to connect using HTTP Proxy, Socket unlike WebClient cannot set proxy with UserName and Password.
My Question is Simple: How to connect using HTTP Proxy with Credentials in Socket?
Please reply only if you have a solution, if you are recommending webclient or other dont reply, I have a strong reason to use Sockets, suggestions for open source libraries, links, tutorials invited.
The username and password for a proxy are sent through the HTTP header.
Use the Proxy-Authorization field in your request header:
Proxy-Authorization: Basic <BASE64("USER:PASS")>
If any of your requests get a response "407 Proxy Authentication Required" you can read the response header field Proxy-Authenticate which will tell you the authentication mode to use when authorizing.
Above it's Basic (the most common one) but there are others like Digest and NTLM. You can read up on the other two here
Your example is showing HTTP Response headers, not HTTP Request headers. What are the HTTP Request headers that you need to send that you cannot?
Doing this directly on a socket is going to be very, very, hard unless you make a bunch of simplifying assumptions (e.g. server never will use chunked encoding or compression, etc). For example, your current code won't work if the server uses keep-alive connections. You'd be far better off using the HTTPWebRequest and using Reflection to tweak any internal members you need.
Another alternative would be to put FiddlerCore into your application (www.fiddler2.com/core). FiddlerCore includes a complete HTTP stack, including support for proxies, compression, chunked-encoding, etc.
加载中,请稍侯......
精彩评论