Assessing File Similarity (Language Agnostic Question)

I have a file store (75,000 files with most files <40 mb and a few multigigabyte video and graphics files) and would like to determine the similarity of the files in it to one another through a simple measure.

I envision it functioning somewhat like a hash key except inverted (a hash key is wildly different for small changes in a file, this number would be slightly different for small changes). The documents would be processed and a number would be generated and these numbers or codes could be compared to find out how similar one document is to another.

Is this possible? Does anyone know of any open source implementation or even algorithm? It would have t开发者_JS百科o cover arbitrary file types so a search solution (like Apache Solr), is not what I'm looking for.

Your are looking for simhashing, e.g. this article.

Please a look in the field of bioinformatics, specifically algorithms used in BLAST and FASTA.

This is a common research task, identifying substrings of a genetic code that are very similar across species.

I can't offer you a solution, but it might be worth taking a look at locality sensitive hashing.