Managing large log files

We have a crawler that persistently crawls our target sites, it's log开发者_开发问答 files are turning out to be quite huge. Over 1 GB in some cases, I'm not too comfortable with deleting or overwriting them. Any examples of how you managed large log files?

Use a cron script to rotate the log files on a daily basis. Basically, you rename your logfile.log to logfile-YYYY-MM-DD.log. This way, instead of one huge logfile, you have smaller ones and are able to find logged messages from a certain time period easily. If you also compress your rotated logs, you will save even more disk space.

You can also use log4net, log4j, or other logging frameworks that automatically change the file names based on file size or date. http://logging.apache.org/log4net/