File integrity check

We have large sets (10+) of very large files (> 1 GB) that we share across our network. There are occasionally a few smaller files too.

Regardless, we want to make sure these files are correct after the transfers. So I am thinking that I could just create a tool that computes a couple of different hashes (SHA1 and MD5, maybe CRC32 too) and saves that in a metadata file. Whoever is on the receiving end can verify the files using the tool.

Should I be concerned that given the size of the files, it is possible that a broken file generates a collision with the original file? I am not worried about someone intentionally generating a collision.

In that case, using bittorrent is an option as well, since it calculates the SHA1 of much smaller chunks (a few KB开发者_如何学Go to a few MB). Another plus is that it will even handle transmission of data. But it also adds an inconvenience, because the same chunk could belong to different files - in which case you need a file's "neighbors" to verify it. Of course I could add dummy padding, but now I am complicating things.

I know 2^80 and even 2^64 are pretty large numbers and that hashes are designed to have an "avalanche effect", so maybe I am just being a little paranoid.

Your best bet might be to use a utility specifically designed for copying large files, like Robocopy or Teracopy.

In addition to verifying file integrity, these utilities also eliminate the annoying cases where copies are aborted due to Windows Explorer file dialogs popping up unexpectedly.

The new version of Ultracopier can do that's too.

The video of presentation show disconnection and resume after disconnection, and the file integrity is correct.

Ultracopier same lot of data-protection mechanism.