Method other than polling to Download a file that may be being updated by server

My code can download any file from a given url. The problem is that i have to download search results that are in .gz file format which is being continuously updated.

my code is downloading incomplete file as the开发者_如何学Python server's search is going on. I have already tried polling, ie i downloaded the file twice and matched the size etc etc.

Is there any other method that can be tried?

It seems you are not properly process response headers - if server returns 'chunked' response, you should load it by parts. To ensure it, user firefox/firebug and check response headers for test request.

See also http://en.wikipedia.org/wiki/Chunked_transfer_encoding

Also check response for other headers like Content-Length/Keep-Alive.