Smart URL Synchronization/Download Engine available open source?

I am wondering if anybody is aware of an open-source URL Synchronization/Download engine or if I will end up writing my own one.

Expected functionality: 开发者_开发技巧- Provide a plain list of HTTP Urls to be synchronized on a local disk store - Engine takes care of synchronizing URL content as effectively (HEAD request to check for changes) and efficiently (all download performed using Gzip compression) - Preferably it would be smart enough to optimize its beahvior by recognizing URLs that change frequently

Is someone aware of an alike-implementation of such engine? Preferably in Perl?

Have you looked at the lwp-mirror program that comes with LWP? That might be a good place to start. Alternatively, wget has more features.