unix file management: best practice for moving files from a staging directory to a final directory

I have a file management process that I need some input on. I'm not 开发者_开发问答sure what's the best approach for this. Here is the basic process flow:

1) Files are created from mulitple servers in:

/dmp/stage/<date>
/dmp/stage/20110329
/dmp/stage/20110330

2) When the file process creation is complete, then copy files from the most current dir /dmp/stage/20110330 to a final directory of /dmp/FINAL

Would it be better to cp the files? or create like a symlink directory for this kind of process? What is a best practice way for this kind of file management? I am trying to do this in either shell or perl.

Thanks.

A symlink could have the advantage that it is clear for an admin or other programmer looking at the system that FINAL is really just a copy/reference to the latest file, as opposed to having data in its own right.

In the interest of completeness, there is another option, a hard link. That way, the contents of the file are shared between the /dmp/stage/xxxx file and the FINAL file (saving space on the disk, if that's an issue) but if the original file gets deleted then the FINAL file will still live on (but it doesn't sound like you intend to delete the stage files.)

It sounds like you are trying to do exactly what rsnapshot does.

rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.

Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals.

Depending on your configuration, it is quite possible to set up in just a few minutes. Files can be restored by the users who own them, without the root user getting involved.

I use rsnapshot to backup my Linux server to a NAS on a daily basis. Because it uses hardlinks I'm able to hold daily, weekly, monthly and quarterly backups with about 1.4x the space of the initial backup. If you wanted to purge files on a weekly basis, then you would make weekly backups your longest hold time and anything that is older than a week automatically gets rotated away.