Update Two Related Files On Disk In a "Secure" Way?

I have two binary files that are related one to another (meaning, when one file's records are updated, the other file's matching records should be updated as well). both files are bin开发者_开发知识库ary files stored on disk.

The updation will look something like this:

UpdateFirstFile() -- first file is updated.....

UpdateSecondFile() -- second file is updated...

what methods should I use to make sure that either BOTH files are updated or NONE of the files is updated?

Both files are flat files (of size 20[MB] each). I know a database would have solved this problem, yet I am note using one due to overhead reasons (every table would require much more than 20[MB] to be stored, and I am short on space and have 1000s of such files...).

Any ideas?

The generic approach would be to implement transactions with some kind of rollback journal.

For example you could use a separate file to record the current contents of each part in each file that will be affected by the update. After your transaction is done, you remove the journal file.

The mere presence of the journal file when starting a transaction would mean that another transaction is either pending or has been interrupted. In that case you use the contents of the journal to reverse any file change that went through before the interruption.

This way you would ensure the atomicity of the update operation. I will leave any other parts of ACID that you need as an exercise to the reader.

Keep in mind that doind this The Right Way is harder than it sounds, especially if you have multiple processes updating the same files.

Do what the RDBMS engines do.

Write an "update sequence number" in each file.

You cannot ever guarantee that both files are written.

However, you can compare the update sequence numbers to see if the files have the same sequence number.

If the sequence numbers disagree, it's logically equivalent to no file having been written. Delete the files and use the backup copies.

If the sequence numbers gree, it's logically equivalent to both having been written.

Both files are flat files (of size 20[MB] each). I know a database would have solved this problem, yet I am note using one due to overhead reasons (every table would require much more than 20[MB] to be stored, and I am short on space and have 1000s of such files...).

You might try HDF5 format (designed to store and organize large amounts of numerical data) to store both datasets in a single file or to store all your data (all 1000s files). It might be simpler than reimplementing database transactions.