Why can't I set Java http User-Agent?

I tried to set the User Agent for http request like this:

public BufferedReader readURL(String url){
        URL urlcon;
        BufferedReader in = null;
        try {
            urlcon = new URL(url);
            connection = (HttpURLConnection)urlcon.openConnection();

            System.setProperty("http.agent", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)");
            connection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)");
            System.out.println(connection.getHeaderField("User-Agent"));
            connection.connect();
            in = new BufferedReader(
                                    new InputStreamReader(
                                        connection.getInputStream()));

            String header = connection.getHeaderField(0);
            System.out.println(header);
            System.out.println("---Start of headers---");
            int i = 1;
            while ((header = connection.getHeaderField(i)) != null) {
                String key = connection.getHeaderFieldKey(i);
                System.out.println(((key==null) ? "" : key + ": ") + header);
                i++;
            }
            System.out.println(connection.getHeaderField("http.agent"));
            System.out.println("---End of headers---");
        } catch (MalformedURLException e) {
            // T开发者_开发百科ODO Auto-generated catch block
            e.printStackTrace();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        return in;
    }

And what i got was User-Agent null:

null

HTTP/1.0 200 OK

---Start of headers---

Server: Apache

Cache-Control: max-age=10

Expires: Sun, 07 Aug 2011 16:09:26 GMT

Vary: Accept-Encoding

Content-Type: text/html

Content-Length: 163582

Date: Sun, 07 Aug 2011 16:09:20 GMT

X-Varnish: 889692780 889684459

Age: 4

Connection: keep-alive

X-Bip: 889692780 70 148

Via: 1.1 CachOS

null

---End of headers---

Why can't I set the User-Agent ?

Use setHeader(), not setRequestProperty.

Setting system property: "http.agent" will change your connection header: "User-Agent", but notice that according to documentation your java version is still written in it:

Misc HTTP properties

http.agent (default: “Java/”) Defines the string sent in the User-Agent request header in http requests.

Note that the string “Java/” will be appended to the one provided in the property (e.g. if -Dhttp.agent=”foobar” is used, the User-Agent header will contain “foobar Java/1.5.0” if the version of the VM is 1.5.0). This property is checked only once at startup.

Notice that property is checked only once at startup, so you may want to first set property, than create your first url and connect.

To verify that appropriate header is sent, you can use "tcpdump". Usage:

tcpdump -n dst host stackoverflow.com -vvvv

The server returns the header information. I would guess "User-Agent" isn't important enough to be return. Just because it isn't returned doesn't mean it isn't sent.

I have a little tool (ieHTTPHeader) that displays the header information. When I do a refresh on this page this it the first set of headers that are sent and returned:

GET /questions/6973981/why-cant-i-set-java-http-user-agent HTTP/1.1
Accept: /
Referer: https://stackoverflow.com/questions/tagged/java?page=2&sort=newest&pagesize=15
Accept-Language: en-ca
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)
Accept-Encoding: gzip, deflate
Host: stackoverflow.com
Connection: Keep-Alive
Cookie: __utmc=140029553; __utma=140029553.1370458634.1310761265.1312727448.1312739618.123; __utmz=140029553.1312739618.123.123.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=yahoo%20finance%20api%20java; __qca=P0-1025379872-1310761265343; m=4; usr=t=cEyCYO7bXECF&s=X6DJTj5kuY8H; __utmb=140029553.15.10.1312739618

HTTP/1.1 200 OK
Cache-Control: public, max-age=60
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Expires: Sun, 07 Aug 2011 18:25:43 GMT
Last-Modified: Sun, 07 Aug 2011 18:24:43 GMT
Vary: *
Date: Sun, 07 Aug 2011 18:24:42 GMT
Content-Length: 12040