Backing up a postgresql data dir with python tarfile. What errors to expect / ignore

I am writing a python script for automating the backing up of the PostgreSQL data directory to enable "Continuous Archiving and Point-In-Time Recovery".

I would like to use Pythons tarfile library to create an archive.

The complication is that the data files can change during the backup. To quote the PostgreSQL manual:

Some file system backup tools emit warnings or errors if the files they are trying to copy change while the copy proceeds. When taking a base backup of an active database, this situation is normal and not an error.开发者_StackOverflow However, you need to ensure that you can distinguish complaints of this sort from real errors.

... some versions of GNU tar return an error code indistinguishable from a fatal error if a file was truncated while tar was copying it. Fortunately, GNU tar versions 1.16 and later exit with 1 if a file was changed during the backup, and 2 for other errors.

When copying the files from the data directory, what exceptions should I be expecting in Python? And how can I determine whether an error is critical?

If you are using the tarfile module, the implementation of this is essentially up to you. GNU tar checks the file size and ctime before and after it processes a file and complains if they differ. It's not an actual error, it's just something that tar feels like it should mention. But in your use case, you don't care about that, so you can just forget about it. The tarfile module certainly won't complain about this by itself.

As far as other exceptions, the possibilities are endless. Pretty much anything that Python throws is probably fatal in this case.

Continuous Archiving is explained in PostgreSQL documentation. It is implemented and there is no need to create additional script - but you will have many files.

24.3.2. Making a Base Backup - follow instructions and you can create tar, data files will not change.

24.3.1. Setting up WAL archiving - here you have to archive new WAL files and they will not change too but they are separate files.

I think it is not safe to add WAL files to base backup. If server crashes while you are adding file to base backup it may become corrupt.