how to handle / read the response with "Transfer-Encoding:chunked"?

May i know how to handle / read the response with "Transfer-Encoding:chunked"?

Currently im using common-httpclient.3.1

My current coding as followings (can handle response with content-length in header only) :-

            httppost = new PostMethod(localurl);
            httppost.setRequestHeader("Content-Type", "application/xml; charset=utf-8");
            RequestEntity entity = new StringRequestEntity(in, "application/xml", "UTF-8");
            httppost.setRequestHeader("Content-length", entity.getContentLength()+"");
            httppost.setRequestEntity(entity);
            for (int i=0; i<retryAttempt; i++) {
                try {
                    httpclient.executeMethod(httppost);
                    if (httppost.getStatusCode() == 200) {

                        br = new BufferedReader(new InputStreamReader(httppost.getResponseBodyAsStream(), httppost.getResponseCharSet()));
                        String reply = null;

                        long len = httppost.getResponseContentLength();

                        if(len != 0) {
                            char[] cbuf = new char[Integer.parseInt(len+"")];
                            if (br.read(cbuf, 0, Integer.parseInt(len+"")) != -1 ) {
                                repOut = String.valueOf(cbuf);
                            }
                        }else{
                            while ((reply = br.readLine()) != null) {
                                if (!reply.equals("")) repOut += reply+" \n ";
                            }
}

Response get from URL (via httppost):

Thread-9, READ: TLSv1 Application Data, length = 443
Padded plaintext after DECRYPTION:  len = 443
0000: 48 54 54 50 2F 31 2E 31   20 32 30 30 20 4F 4B 0D  HTTP/1.1 200 OK.
0010: 0A 53 65 72 76 65 72 3A   20 53 54 52 4F 4E 47 48  .Server: STRONGH
0020: 4F 4C 44 2F 33 2E 30 2F   4F 70 65 6E 42 53 44 0D  OLD/3.0/OpenBSD.
0030: 0A 44 61 74 65 3A 20 53   75 6E 2C 20 32 32 20 4D  .Date: Sun, 22 M
0040: 61 79 20 32 30 31 31 20   31 34 3A 30 31 3A 30 38  ay 2011 14:01:08
0050: 20 47 4D 54 0D 0A 43 6F   6E 74 65 6E 74 2D 74 79   GMT..Content-ty
0060: 70 65 3A 20 61 70 70 6C   69 63 61 74 69 6F 6E 2F  pe: application/
0070: 78 6D 6C 3B 63 68 61 72   73 65 74 3D 55 54 46 2D  xml;charset=UTF-
0080: 38 0D 0A 53 65 74 2D 63   6F 6F 6B 69 65 3A 20 4A  8..Set-cookie: J
0090: 53 45 53 53 49 4F 4E 49   44开发者_开发问答 3D 34 34 42 35 33 45  SESSIONID=44B53E
00A0: 35 46 38 30 39 41 35 35   32 34 32 41 44 36 34 38  5F809A55242AD648
00B0: 33 41 39 33 32 42 30 46   44 36 3B 20 50 61 74 68  3A932B0FD6; Path
00C0: 3D 2F 3B 20 53 65 63 75   72 65 0D 0A 54 72 61 6E  =/; Secure..Tran
00D0: 73 66 65 72 2D 65 6E 63   6F 64 69 6E 67 3A 20 63  sfer-encoding: c
00E0: 68 75 6E 6B 65 64 0D 0A   0D 0A 62 62 0D 0A 3C 3F  hunked....bb..<?
00F0: 78 6D 6C 20 76 65 72 73   69 6F 6E 3D 22 31 2E 30  xml version="1.0
0100: 22 20 65 6E 63 6F 64 69   6E 67 3D 22 55 54 46 2D  " encoding="UTF-
0110: 38 22 3F 3E 0A 3C 54 68   72 65 65 44 53 65 63 75  8"?>.<ThreeDSecu
0120: 72 65 3E 3C 4D 65 73 73   61 67 65 20 69 64 3D 22  re><Message id="
0130: 50 41 54 72 61 6E 73 52   65 71 53 70 65 63 34 31  PATransReqSpec41
0140: 37 39 32 30 31 31 30 35   32 32 32 32 30 30 30 32  7920110522220002
0150: 36 30 31 37 32 39 33 33   37 36 38 36 35 22 3E 3C  6017293376865"><
0160: 50 41 54 72 61 6E 73 52   65 73 3E 3C 76 65 72 73  PATransRes><vers
0170: 69 6F 6E 3E 31 2E 30 2E   32 3C 2F 76 65 72 73 69  ion>1.0.2</versi
0180: 6F 6E 3E 3C 2F 50 41 54   72 61 6E 73 52 65 73 3E  on></PATransRes>
0190: 3C 2F 4D 65 73 73 61 67   65 3E 3C 2F 54 68 72 65  </Message></Thre
01A0: 65 44 53 65 63 75 72 65   3E 0D 0A 76 67 28 7C B3  eDSecure>..vg(..
01B0: C8 F9 7C A1 C9 77 2F F6   E8 8A 42                 .....w/...B
[Raw read]: length = 5

commons-httpclient will de-chunk your server's response out of the box, there's no need for you to handle chunking. However, you may be running into problems because - due to the nature of chunking - you don't know the length of the content until you've read it all. (Requesting the content length will return -1.) You therefore cannot allocate a fixed-size buffer to hold the entire content.

If you change your test of the getResponseContentLength return value to "if(len > 0)", I think you'll see that you fall into the readLine case when you encounter a response that is chunked, and that it should read the entire response line-by-line.

Another library to consider is the Jodd Http library (http://jodd.org/doc/http.html). It handles Chunked transfer encoding for you and you get the whole body as a string back.

The nice thing is that the API is really simple:

HttpRequest httpRequest = HttpRequest.get("http://jodd.org");
HttpResponse response = httpRequest.send();

For some reason I was having a similiar error. I was using jackson deserealizer to convert response.getEntity().getContent() using an objectMapper to a custom object. The server was returning a chunked response and the InputStream returned had a lenght longer than the content. This originated an exception. I resolved by readAllBytes() of the InputStream.