http server not honoring Accept-Encoding: gzip unless User-Agent is known browser

When making an HTTP request, I set Accept-Encoding: gzip. However, I notice that the server doesn't actually compress the response unless User-Agent is "well-known". I have the following Android code that demonstrates the problem, it gets "http://www.google.com" using my awesome user-agent, and checks if the response is compressed:

    HttpParams params = new BasicHttpParams();
    HttpProtocolParams.setUserAgent(params, "MyApp/1.0 (Awesome)");

    DefaultHttpClient client = new DefaultHttpClient(params);       
    HttpGet request = new HttpGet("http://www.google.com");
    request.setHeader("Accept-Encoding", "gzip");

    HttpResponse response = client.execute(request);
    HttpEntity entity = response.getEntity();
    Header header = entity.getContentEncoding();
    boolean isCompressed = false;
    if (header != null) {
        HeaderElement[] codecs = header.getElements();
        for (int i = 0; i < codecs.length; i++) {
            if (codecs[i].getName().equalsIgnoreCase("gzip")) {
                isCompressed = true;
                break;
            }
        }
    }

    if (isCompressed)
        Log.i(TAG, "IT IS COMPRESSED");
    else
        Log.i(TAG, "IT IS NOT COMPRESSED");

Running this code prints out "IT IS NOT COMPRESSED" :(

But if I set the user agent to "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0", it outputs "IT IS COMPRESSED".

I know it is the server's choice whether to actually compress or not, but why is it choosing based on "User-Agent"? Is there any other header options I should be sending to coax the server to compress, irrespective of the use开发者_开发技巧r-agent?

EDIT:

I know there's no proxy/etc along the way that is mangling the Accept-Encoding header, because when I get http://1.cgi.browserscope.net/cgi-bin/resource.cgi?headers=1, I see the Accept-Encoding header is received correctly

It's almost certainly doing this because some legacy User-Agents advertise support for GZIP and then fail to properly decompress the content. For that reason, some server-side frameworks are known to only send compressed responses to clients known not to suffer bugs like that.