Sort results from find and grab subset (bash)
I have a pair of bash scripts, 1 that dumps mysql dbs, and the second to purge old backups.
I have always relied on date, so any files older than 7 days are purged, and new backups created daily. The result was a set of backups 7 days back.
Well now that I have 20+ dbs I have separated that daily job into a daily, and weekly job. If I continue to use the find and date method I would lose the past week.
QUESTION
How can I sort the output of a find by date, and then purge all but the most recent 7. (so even if the most recent is 2 months old, it remains)
This is as far as I got, but it is not as expected. Perhaps using ls might be the better choice 9find is a remnant from using date as the sole criteria for purging)
OLDFILES=`find -regex .*sql.*`
set -- $OLDFILES
if [ -z $1 ]
then
printf "\tNo files matching purge criteria\n" | tee -a $LOGFILE
else
printf "\tSQL Files being Delete from $HERE\n" | tee -a $LOGFILE
printf "\t\t%s\n" $OLDFILES | tee -a $LOGFILE
fi
# $RETAIN is the # of recent to keep
# so I want to grab a subset from 0 to (total - retain), and purge them
TOBURN=$(( $# - $RETAIN ))
printf " grab first %s files. %s - %s" $TOBURN $# $RETAIN
if [ $TOBURN -gt 0 ]
then
TOPURGE=("$@:0:$TOBURN")
printf "\n\nREMOVING..\n\t\t%s\n" $TOPURGE
fi
Existing files
./webbmaster_bellarose_joomla_01-09-2010.sql.gz
./webbmaster_bellarose_joomla_12-25-2009.sql.gz
./webbmaster_bellarose_joomla_01-10-2010.sql.gz
./webbmaster_bellarose_joomla_01-04-2010.sql.gz
./webbmaster_bellarose_joomla_01-25-2010.sql.gz
./webbmaster_bellarose_joomla_01-29-2010.sql.gz
./webbmaster_bellarose_joomla_12-30-2009.sql.gz
./webbmaster_bellarose_joomla_12-28-2009.sql.gz
./webbmaster_bellarose_joomla_01-06-2010.sql.gz
./webbmaster_bellarose_joomla_01-13-2010.sql.gz
./webbmaster_bellarose_joomla_01-24-2010.sql.gz
./webbmaster_bellarose_joomla_12-21-2009.sql.gz
./webbmaster_bellarose_joomla_12-24-2009.sql.gz
./webbmaster_bellarose_joomla_12-27-2009.sql.gz
./webbmaster_bellarose_joomla_01-26-2010.sql.gz
./webbmaster_bellarose_joomla_12-17-2009.sql.gz
./webbmaster_bellarose_joomla_01-03-2010.sql.gz
./webbmaster_bellarose_joomla_01-21-2010.sql.gz
./webbmaster_bellarose_joomla_01-20-2010.sql.gz
./webbmaster_bellarose_joomla_01-16-2010.sql.gz
./webbmaster_bellarose_joomla_12-31-2009.sql.gz
./webbmaster_bellarose_joomla_01-11-2010.sql.gz
./webbmaster_bellarose_joomla_01-05-2010.sql.gz
./webbmaster_bellarose_joomla_01-14-2010.sql.gz
./webbmaster_bellarose_joomla_01-22-2010.sql.gz
./webbmaster_bellarose_joomla_01-12-2010.sql.gz
./webbmaster_bellarose_joomla_01-17-2010.sql.gz
./webbmaster_bellarose_joomla_12-18-2009.sql.gz
./webbmaster_bellarose_joomla_12-20-2009.sql.gz
./webbmaster_bellarose_joomla_01-08-2010.sql.gz
./webbmaster_bellarose_joomla_12-22-2009.sql.gz
./webbmaster_bellarose_joomla_01-27-2010.sql.gz
./webbmaster_bellarose_joomla_12-23-2009.sql.gz
./webbmaster_bellarose_joomla_01-07-2010.sql.gz
./webbmaster_bellarose_joomla_12-26-2009.sql.gz
./webbmaster_bellarose_joomla_01-19-2010.sql.gz
./webbmaster_bellarose_joomla_12-29-2009.sql.gz
./webbmaster_bellarose_joomla_01-15-2010.sql.gz
./webbmaster_bellarose_joomla_01-28-2010.sql.gz
./webbmaster_bellarose_joomla_01-01-2010.sql.gz
./webbmaster_bellarose_joomla_01-18-2010.sql.gz
./webbmaster_bellarose_joomla_01-02-2010.sql.gz
./webbmaster_bellarose_joomla_01-23-2010.sql.gz
./webbmaster_bellarose_joomla_12-19-2009.sql.gz
Targets for purging (which I want to set in va开发者_如何学编程riable for simple rm -f $var)
./webbmaster_bellarose_joomla_01-09-2010.sql.gz
./webbmaster_bellarose_joomla_12-25-2009.sql.gz
./webbmaster_bellarose_joomla_01-10-2010.sql.gz
./webbmaster_bellarose_joomla_01-04-2010.sql.gz
./webbmaster_bellarose_joomla_12-30-2009.sql.gz
./webbmaster_bellarose_joomla_12-28-2009.sql.gz
./webbmaster_bellarose_joomla_01-06-2010.sql.gz
./webbmaster_bellarose_joomla_01-13-2010.sql.gz
./webbmaster_bellarose_joomla_12-21-2009.sql.gz
./webbmaster_bellarose_joomla_12-24-2009.sql.gz
./webbmaster_bellarose_joomla_12-27-2009.sql.gz
./webbmaster_bellarose_joomla_12-17-2009.sql.gz
./webbmaster_bellarose_joomla_01-03-2010.sql.gz
./webbmaster_bellarose_joomla_01-21-2010.sql.gz
./webbmaster_bellarose_joomla_01-20-2010.sql.gz
./webbmaster_bellarose_joomla_01-16-2010.sql.gz
./webbmaster_bellarose_joomla_12-31-2009.sql.gz
./webbmaster_bellarose_joomla_01-11-2010.sql.gz
./webbmaster_bellarose_joomla_01-05-2010.sql.gz
./webbmaster_bellarose_joomla_01-14-2010.sql.gz
./webbmaster_bellarose_joomla_01-22-2010.sql.gz
./webbmaster_bellarose_joomla_01-12-2010.sql.gz
./webbmaster_bellarose_joomla_01-17-2010.sql.gz
./webbmaster_bellarose_joomla_12-18-2009.sql.gz
./webbmaster_bellarose_joomla_12-20-2009.sql.gz
./webbmaster_bellarose_joomla_01-08-2010.sql.gz
./webbmaster_bellarose_joomla_12-22-2009.sql.gz
./webbmaster_bellarose_joomla_12-23-2009.sql.gz
./webbmaster_bellarose_joomla_01-07-2010.sql.gz
./webbmaster_bellarose_joomla_12-26-2009.sql.gz
./webbmaster_bellarose_joomla_01-19-2010.sql.gz
./webbmaster_bellarose_joomla_12-29-2009.sql.gz
./webbmaster_bellarose_joomla_01-15-2010.sql.gz
./webbmaster_bellarose_joomla_01-01-2010.sql.gz
./webbmaster_bellarose_joomla_01-18-2010.sql.gz
./webbmaster_bellarose_joomla_01-02-2010.sql.gz
./webbmaster_bellarose_joomla_12-19-2009.sql.gz
(dates 1/23 - 1/29 are left intact)
Please do yourself (and everyone else) a big favour and start writing dates in ISO-8601 format (i.e. YYYY-MM-DD).
E.g. instead of
./webbmaster_bellarose_joomla_12-29-2009.sql.gz
you would have
./webbmaster_bellarose_joomla_2009-12-29.sql.gz
Doing so have many benefits, most importantly here that chronological and alphabetical order becomes identical. No need for find, just list the files, reverse order, delete the first 7 lines and delete any remaining files:
ls *sql* | tac | sed 1,7d | tr '\012' '\000' | xargs -0 --no-run-if-empty rm
Update: What do you mean by "moving forward"? As fas as I can tell my solution solves your problem 100%:
/tmp/so>cat existing_files
./webbmaster_bellarose_joomla-2009-12-17.sql.gz
./webbmaster_bellarose_joomla-2009-12-18.sql.gz
./webbmaster_bellarose_joomla-2009-12-19.sql.gz
./webbmaster_bellarose_joomla-2009-12-20.sql.gz
./webbmaster_bellarose_joomla-2009-12-21.sql.gz
./webbmaster_bellarose_joomla-2009-12-22.sql.gz
./webbmaster_bellarose_joomla-2009-12-23.sql.gz
./webbmaster_bellarose_joomla-2009-12-24.sql.gz
./webbmaster_bellarose_joomla-2009-12-25.sql.gz
./webbmaster_bellarose_joomla-2009-12-26.sql.gz
./webbmaster_bellarose_joomla-2009-12-27.sql.gz
./webbmaster_bellarose_joomla-2009-12-28.sql.gz
./webbmaster_bellarose_joomla-2009-12-29.sql.gz
./webbmaster_bellarose_joomla-2009-12-30.sql.gz
./webbmaster_bellarose_joomla-2009-12-31.sql.gz
./webbmaster_bellarose_joomla-2010-01-01.sql.gz
./webbmaster_bellarose_joomla-2010-01-02.sql.gz
./webbmaster_bellarose_joomla-2010-01-03.sql.gz
./webbmaster_bellarose_joomla-2010-01-04.sql.gz
./webbmaster_bellarose_joomla-2010-01-05.sql.gz
./webbmaster_bellarose_joomla-2010-01-06.sql.gz
./webbmaster_bellarose_joomla-2010-01-07.sql.gz
./webbmaster_bellarose_joomla-2010-01-08.sql.gz
./webbmaster_bellarose_joomla-2010-01-09.sql.gz
./webbmaster_bellarose_joomla-2010-01-10.sql.gz
./webbmaster_bellarose_joomla-2010-01-11.sql.gz
./webbmaster_bellarose_joomla-2010-01-12.sql.gz
./webbmaster_bellarose_joomla-2010-01-13.sql.gz
./webbmaster_bellarose_joomla-2010-01-14.sql.gz
./webbmaster_bellarose_joomla-2010-01-15.sql.gz
./webbmaster_bellarose_joomla-2010-01-16.sql.gz
./webbmaster_bellarose_joomla-2010-01-17.sql.gz
./webbmaster_bellarose_joomla-2010-01-18.sql.gz
./webbmaster_bellarose_joomla-2010-01-19.sql.gz
./webbmaster_bellarose_joomla-2010-01-20.sql.gz
./webbmaster_bellarose_joomla-2010-01-21.sql.gz
./webbmaster_bellarose_joomla-2010-01-22.sql.gz
./webbmaster_bellarose_joomla-2010-01-23.sql.gz
./webbmaster_bellarose_joomla-2010-01-24.sql.gz
./webbmaster_bellarose_joomla-2010-01-25.sql.gz
./webbmaster_bellarose_joomla-2010-01-26.sql.gz
./webbmaster_bellarose_joomla-2010-01-27.sql.gz
./webbmaster_bellarose_joomla-2010-01-28.sql.gz
./webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so>cat targets_for_purging
./webbmaster_bellarose_joomla-2009-12-17.sql.gz
./webbmaster_bellarose_joomla-2009-12-18.sql.gz
./webbmaster_bellarose_joomla-2009-12-19.sql.gz
./webbmaster_bellarose_joomla-2009-12-20.sql.gz
./webbmaster_bellarose_joomla-2009-12-21.sql.gz
./webbmaster_bellarose_joomla-2009-12-22.sql.gz
./webbmaster_bellarose_joomla-2009-12-23.sql.gz
./webbmaster_bellarose_joomla-2009-12-24.sql.gz
./webbmaster_bellarose_joomla-2009-12-25.sql.gz
./webbmaster_bellarose_joomla-2009-12-26.sql.gz
./webbmaster_bellarose_joomla-2009-12-27.sql.gz
./webbmaster_bellarose_joomla-2009-12-28.sql.gz
./webbmaster_bellarose_joomla-2009-12-29.sql.gz
./webbmaster_bellarose_joomla-2009-12-30.sql.gz
./webbmaster_bellarose_joomla-2009-12-31.sql.gz
./webbmaster_bellarose_joomla-2010-01-01.sql.gz
./webbmaster_bellarose_joomla-2010-01-02.sql.gz
./webbmaster_bellarose_joomla-2010-01-03.sql.gz
./webbmaster_bellarose_joomla-2010-01-04.sql.gz
./webbmaster_bellarose_joomla-2010-01-05.sql.gz
./webbmaster_bellarose_joomla-2010-01-06.sql.gz
./webbmaster_bellarose_joomla-2010-01-07.sql.gz
./webbmaster_bellarose_joomla-2010-01-08.sql.gz
./webbmaster_bellarose_joomla-2010-01-09.sql.gz
./webbmaster_bellarose_joomla-2010-01-10.sql.gz
./webbmaster_bellarose_joomla-2010-01-11.sql.gz
./webbmaster_bellarose_joomla-2010-01-12.sql.gz
./webbmaster_bellarose_joomla-2010-01-13.sql.gz
./webbmaster_bellarose_joomla-2010-01-14.sql.gz
./webbmaster_bellarose_joomla-2010-01-15.sql.gz
./webbmaster_bellarose_joomla-2010-01-16.sql.gz
./webbmaster_bellarose_joomla-2010-01-17.sql.gz
./webbmaster_bellarose_joomla-2010-01-18.sql.gz
./webbmaster_bellarose_joomla-2010-01-19.sql.gz
./webbmaster_bellarose_joomla-2010-01-20.sql.gz
./webbmaster_bellarose_joomla-2010-01-21.sql.gz
./webbmaster_bellarose_joomla-2010-01-22.sql.gz
/tmp/so>sed 's/\(.\{29\}\)_\(..-..\)-\(....\)/\1-\3-\2/' existing_files | sort > existing_files.8601
/tmp/so>sed 's/\(.\{29\}\)_\(..-..\)-\(....\)/\1-\3-\2/' targets_for_purging | sort > targets_for_purging.8601
/tmp/so>mkdir files
/tmp/so>cd files
/tmp/so/files>xargs touch < ../existing_files.8601
/tmp/so/files>ls -1
webbmaster_bellarose_joomla-2009-12-17.sql.gz
webbmaster_bellarose_joomla-2009-12-18.sql.gz
webbmaster_bellarose_joomla-2009-12-19.sql.gz
webbmaster_bellarose_joomla-2009-12-20.sql.gz
webbmaster_bellarose_joomla-2009-12-21.sql.gz
webbmaster_bellarose_joomla-2009-12-22.sql.gz
webbmaster_bellarose_joomla-2009-12-23.sql.gz
webbmaster_bellarose_joomla-2009-12-24.sql.gz
webbmaster_bellarose_joomla-2009-12-25.sql.gz
webbmaster_bellarose_joomla-2009-12-26.sql.gz
webbmaster_bellarose_joomla-2009-12-27.sql.gz
webbmaster_bellarose_joomla-2009-12-28.sql.gz
webbmaster_bellarose_joomla-2009-12-29.sql.gz
webbmaster_bellarose_joomla-2009-12-30.sql.gz
webbmaster_bellarose_joomla-2009-12-31.sql.gz
webbmaster_bellarose_joomla-2010-01-01.sql.gz
webbmaster_bellarose_joomla-2010-01-02.sql.gz
webbmaster_bellarose_joomla-2010-01-03.sql.gz
webbmaster_bellarose_joomla-2010-01-04.sql.gz
webbmaster_bellarose_joomla-2010-01-05.sql.gz
webbmaster_bellarose_joomla-2010-01-06.sql.gz
webbmaster_bellarose_joomla-2010-01-07.sql.gz
webbmaster_bellarose_joomla-2010-01-08.sql.gz
webbmaster_bellarose_joomla-2010-01-09.sql.gz
webbmaster_bellarose_joomla-2010-01-10.sql.gz
webbmaster_bellarose_joomla-2010-01-11.sql.gz
webbmaster_bellarose_joomla-2010-01-12.sql.gz
webbmaster_bellarose_joomla-2010-01-13.sql.gz
webbmaster_bellarose_joomla-2010-01-14.sql.gz
webbmaster_bellarose_joomla-2010-01-15.sql.gz
webbmaster_bellarose_joomla-2010-01-16.sql.gz
webbmaster_bellarose_joomla-2010-01-17.sql.gz
webbmaster_bellarose_joomla-2010-01-18.sql.gz
webbmaster_bellarose_joomla-2010-01-19.sql.gz
webbmaster_bellarose_joomla-2010-01-20.sql.gz
webbmaster_bellarose_joomla-2010-01-21.sql.gz
webbmaster_bellarose_joomla-2010-01-22.sql.gz
webbmaster_bellarose_joomla-2010-01-23.sql.gz
webbmaster_bellarose_joomla-2010-01-24.sql.gz
webbmaster_bellarose_joomla-2010-01-25.sql.gz
webbmaster_bellarose_joomla-2010-01-26.sql.gz
webbmaster_bellarose_joomla-2010-01-27.sql.gz
webbmaster_bellarose_joomla-2010-01-28.sql.gz
webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so/files>ls *sql* | tac | sed 1,7d | tr '\012' '\000' | xargs -0 --no-run-if-empty rm
/tmp/so/files>ls -1 ./*
./webbmaster_bellarose_joomla-2010-01-23.sql.gz
./webbmaster_bellarose_joomla-2010-01-24.sql.gz
./webbmaster_bellarose_joomla-2010-01-25.sql.gz
./webbmaster_bellarose_joomla-2010-01-26.sql.gz
./webbmaster_bellarose_joomla-2010-01-27.sql.gz
./webbmaster_bellarose_joomla-2010-01-28.sql.gz
./webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so/files>comm -3 ../existing_files.8601 ../targets_for_purging.8601
./webbmaster_bellarose_joomla-2010-01-23.sql.gz
./webbmaster_bellarose_joomla-2010-01-24.sql.gz
./webbmaster_bellarose_joomla-2010-01-25.sql.gz
./webbmaster_bellarose_joomla-2010-01-26.sql.gz
./webbmaster_bellarose_joomla-2010-01-27.sql.gz
./webbmaster_bellarose_joomla-2010-01-28.sql.gz
./webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so/files>
If the files are all in one directory (i.e. you do not need to use find to recurse though subdirectories), you can use the -t argument to ls(1) to order the file list by time and then remove the first 7 entries with tail(1).
rm -f $(ls -t *sql* | tail -n +8)
[Edit: I added -f in case the substitution results in an empty list]
or in a loop...
ls -t *sql* | tail -n +8 | while read file
echo "Purging: $file"
rm "$file"
done
The argument to tail is +8 because you want it to start tailing from the 8th line onwards (i.e. drop the first 7).
After removing the files, you can just burn what's left.
OLDIFS="$IFS"
IFS=$'\n'
topurge=($(find -mindepth 1 -printf "%T@ %P\n" | sort -n -r | cut -d' ' -f 2- | tail -n +7))
IFS="$OLDIFS"
Here is one thought:
oldest_to_keep=`find . -name \*.sql | xargs \ls -1rc | tail -7 | head -1`
find . -name \*.sql -not -samefile $oldest_to_keep -not -newer $oldest_to_keep
Where the second find ought to have -exec rm \{\} ;
if you've convinced yourself this is sane. :)
精彩评论