Why does git hash-object return a different hash than openssl sha1?

Context: I downloaded a file (Audirvana 0.7.1开发者_开发百科.zip) from code.google to my Macbook Pro (Mac OS X 10.6.6).

I wanted to verify the checksum, which for that particular file is posted as 862456662a11e2f386ff0b24fdabcb4f6c1c446a (SHA-1). git hash-object gave me a different hash, but openssl sha1 returned the expected 862456662a11e2f386ff0b24fdabcb4f6c1c446a.

The following experiment seems to rule out any possible download corruption or newline differences and to indicate that there are actually two different algorithms at play:

$ echo A > foo.txt
$ cat foo.txt
A
$ git hash-object foo.txt 
f70f10e4db19068f79bc43844b49f3eece45c4e8
$ openssl sha1 foo.txt 
SHA1(foo.txt)= 7d157d7c000ae27db146575c08ce30df893d3a64

What's going on?

You see a difference because git hash-object doesn't just take a hash of the bytes in the file - it prepends the string "blob " followed by the file size and a NUL to the file's contents before hashing. There are more details in this other answer on Stack Overflow:

• How to assign a Git SHA1's to a file without Git?

Or, to convince yourself, try something like:

$ echo -n hello | git hash-object --stdin
b6fc4c620b67d95f953a5c1c1230aaab5db5a1b0

$ printf 'blob 5\0hello' > test.txt
$ openssl sha1 test.txt
SHA1(test.txt)= b6fc4c620b67d95f953a5c1c1230aaab5db5a1b0

The SHA1 digest is calculated over a header string followed by the file data. The header consists of the object type, a space and the object length in bytes as decimal. This is separated from the data by a null byte.

So:

$ git hash-object foo.txt
f70f10e4db19068f79bc43844b49f3eece45c4e8
$ ( perl -e '$size = (-s shift); print "blob $size\x00"' foo.txt \
               && cat foo.txt ) | openssl sha1
f70f10e4db19068f79bc43844b49f3eece45c4e8

One consequence of this is that "the" empty tree and "the" empty blob have different IDs. That is:

e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 always means "empty file" 4b825dc642cb6eb9a060e54bf8d69288fbee4904 always means "empty directory"

You will find that you can in fact do git ls-tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904 in a new git repository with no objects registered, because it is recognised as a special case and never actually stored (with modern Git versions). By contrast, if you add an empty file to your repo, a blob "e69de29bb2d1d6434b8b29ae775ad8c2e48c5391" will be stored.

Git stores objects as [Object Type, Object Length, delimeter (\0), Content] In your case:

$ echo "A" | git hash-object --stdin
f70f10e4db19068f79bc43844b49f3eece45c4e8

Try to calculate hash as:

$ echo -e "blob 2\0A" | shasum 
f70f10e4db19068f79bc43844b49f3eece45c4e8  -

Note using -e (for bash shell) and adjusting length for newline.

The answer lies here:

How to assign a Git SHA1's to a file without Git?

git calculates on file metadata + contents, not just contents.

That is a good enough answer for now, and the takeaway is that git is not the tool for checksumming downloads.

Take care to filters !

git is actually filtering the file before calculating the sha. Typically \r\n end of lines are converted to \n. this is why you may have different results between git hash-object and git hash-object --no-filters some other stuff may be filtered and .gitattributes can have an impact on the results.

little example using windows cmd :

create test files in a new folder:

$ echo this is a test $Id$ > test1.txt
$ echo this is a test $Id: ffbf88668784c14e809c8c449d799b654d7a5fc5 $ > test2.txt

now use git hash-object

$ git hash-object test1.txt
0c3a75d8155d54c2367e290cf7f33434805410be

$ git hash-object test2.txt
60fff1b8ec47ed41254719681e32369d640d6a0f

$ git hash-object --no-filters test2.txt
2f68d9b80a38fb800f039ef9062c764d2a4d4352

different files leads to different hashes : OK but git does somehow filter the file as --no-filters has an impact.

now create a git repo and .gitattributes in the folder:

$ git init .
Initialized empty Git repository in ~/.git

$ echo *.txt ident > .gitattributes

$ git hash-object test1.txt
0c3a75d8155d54c2367e290cf7f33434805410be

$ git hash-object test2.txt
0c3a75d8155d54c2367e290cf7f33434805410be

$ git hash-object --no-filters test2.txt
2f68d9b80a38fb800f039ef9062c764d2a4d4352

Now test1 and test2 have the same hash ! but --no-filters option is still giving the same value.

Conclusion: you can get the same hash with git and openssl but you need to make sure that your file is not impacted by git filters.