Generate a value that identify a string and find duplicates / nearly duplicates

I need a table that contains information about a file, and a value that identifies the content in this file. The filetypes is primarily pdf, and i have the logic to extract content from the files in place.

Right now i create a hash value of the content and place this in the table. With this value i am able to find duplicate files. But, the customer also want to identify files that are nearly 开发者_如何学运维the same. For example if the copyright information is changed - or some other minor change. There is going to be a manual review of the result of this, so I don't need a 100% hit rate.

The first thing that come to my mind is to make a table that contains the 50 most popular words, and associate them with an id. Than i can use this and do a word count in each of the documents. The result will be a string like this ( where word 1 is hit 20 times, word 2 12 times ) "1:20-2:12......"

Anyone have a better suggestion on how to make a value that can identify nearly identical files / strings?

You may take a look at the Levenshtein distance which is used to compare similarity between sequences and here's a sample implementation in C#.

Two solutions come to my mind:
1. There are many diff tools that allow the text to be compared, like WinMerge, which can also compare whole folders. You could extract pdf contents to text files, then invoke the tool from inside your program and count the number of differences, which should tell you how different the files are.
2. You are probably computing md5 or sha1 hashes, which change drastically on minor change of the input. Try to find (or create) algorithm that doesn't change the hash that much. You may have many collisions, but this should be easy to tackle.