compression (header) bug in .NET HttpWebRequest?

I just noticed that using the .NET HttpWebRequest, I wasn't getting gzip'd contents back from the server (running nginx 0.8.52). Testing with curl, I verified that gzip was set up properly server-side, then dug into the issue with the VS debugger.

The culprit turns out to be how the Accept-Encoding header field is generated: if I set

 request.AutomaticDecompression = DecompressionMethods.GZip | 
                                   DecompressionMethods.Deflate`

I get the following header:

`Accept-Encoding: gzip, deflate,gzip, deflate`. 

If I set

request.AutomaticDecompression = DecompressionMethods.GZip; 

I get

Accept-Encoding: gzip,gzip

I haven't checked what the HTTP specification says, but nginx ought to handle the situation, but instead it returns Vary: Accept-Encoding back. On the other hand, the Accept-Encoding header generated by HttpWebRequest definitely doesn't look right, and seems like a bug to me.

If I don't specify AutomaticDecompression and manually set the Accept-Encoding header, I get gzip'd content back, but HttpWebRequest seems to be pretty dumb, and doesn't decode the compressed stream (I guess it relies on an internal IsCompressed flag rather than parsing the response headers).

Any suggestions?

EDIT:

It should be noted that there's authentication involved; I've just found out that HttpWebRequest initially does a request without including the supplied credentials. For this request, the Accept-Encoding header is specified correctly. The duplication occurs after getting the 401 server response and re-doing the request with开发者_如何学编程 credentials.

EDIT 2:

I posted a Microsoft Connect bug report.

Seems to be a bug alright. Possible workarounds:

1. Don't use AutomaticDecompression, set "Accept-Encoding" header field manually, and manually handle decompression of the response stream if "Content-Encoding" property is set.
2. Don't use Credentials property, instead manually construct HTTP Authorization header. This gets rid of unnecessary server roundtrip and fixes the header field duplication bug.

LOL, funny you mention this, I noted this just on Friday ;P

I have not seen any issues due to this, my response is still compressed.

I have also noted the several proxy requests. But I cannot for the life of me get GZip compression working over MS ISA or the new one with the long name server. It seems to just strip the request header, and the server does not even know about it. I am having a meeting with one of our network admins tomorrow to see if the issue is proxy related.

Update:

I should note that, I have no problem placing a Squid proxy between myself and server, but that was simply using basic authentication, and not the fancy challenge-response stuff Windows/ISA uses.

Update 2:

I forgot to mention, the Accept-Encoding header 'disappears' too when using a browser to access the site. This is why I think the real problem might not be specific to .NET. Testing continues.

Update 3:

I did a capture using FF3.6 going via ISA proxy and pretty much the same result. Going via ISA, only does a 'single' proxy authenticate, but the final response is still not compressed.

Given the high rate of hereditary bugs in MS products, I would have to side with ISA/FTMG being the culprit here.

Update 4 (solved):

I would strongly advise to look into ISA/FTMG configuration, if that is used in the network hosting the server.

Update 5 (solved, really):

Tried another setting, and that did the 'trick'. On the FTMG proxy, we had to enable HTTP compression for the 'External' network, as well as make sure the compressible content-types were correct (had to add application/soap+xml for WCF). This allows the proxy to do compression, and the actual web server does not, which may or may not be ideal, but at least it works. :)

I would strongly advise to look into ISA/FTMG configuration, if that is used in the network hosting the server.

Capture log:

Here is my trace hitting the site with IE8 via Forefront TMG proxy captured with Wireshark. If the bug is .NET specific, I would expect the result to be different, but from what I can see, it behaves the same.

Initial request:

GET http://www.foo.com/ClientInstall HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: www.foo.com

HTTP/1.1 407 Proxy Authentication Required 
  ( Forefront TMG requires authorization to fulfill the request. 
  Access to the Web Proxy filter is denied.  )
Via: 1.1 PROXP01
Proxy-Authenticate: Negotiate
Proxy-Authenticate: Kerberos
Proxy-Authenticate: NTLM
Proxy-Authenticate: Basic realm="PROXP01.bar.com"
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 4140  

Authenticate with proxy, but fails?:

GET http://www.foo.com/ClientInstall HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: www.foo.com
Proxy-Authorization: Negotiate TlRMTVNTUAABAAAA...

HTTP/1.1 407 Proxy Authentication Required ( Access is denied.  )
Via: 1.1 PROXP01
Proxy-Authenticate: Negotiate TlRMTVNTUAACAAAAB...
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 0     

Final sequence:

GET http://www.foo.com/ClientInstall HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Proxy-Authorization: Negotiate TlRMTVNTUAADAAAAGAAYAH4AAAA...
Host: www.foo.com

HTTP/1.1 301 OK
Via: 1.1 PROXP01
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 160
Date: Wed, 24 Nov 2010 04:34:31 GMT
Age: 308
Location: http://www.foo.com/ClientInstall/
Content-Type: text/html; charset=UTF-8
Server: Microsoft-IIS/7.0
X-Powered-By: ASP.NET

GET http://www.foo.com/ClientInstall/ HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: www.foo.com

HTTP/1.1 200 OK
Via: 1.1 PROXP01
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 4621
Date: Wed, 24 Nov 2010 04:39:39 GMT
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.0
Cache-Control: private
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET