I’m shuffling a few 1TB images from a Ubuntu machine to a Windows box. Thinking I’d be slick, I used sha1sum to calculate the SHA1 hash of the first image before sending.
Then I used the File Checksum Integrity Verifier (fciv.exe) by Microsoft to do the same once I had used ncftp to transfer the image to the Windows box.
fciv.exe -sha1 system-image.vdk
But I ran into an immediate problem. The hashes being returned were not the same.
Is it ncftp?
I tried it using regular ftp. Same deal.
Is it an encoding issue?
I’ve been testing with text files, which leaves some room for differing interpretations by the OS. So let’s play with a binary again downloaded via ftp.
(HashCheck for Windows) File: fciv.exe CRC-32: 71cbdfd9 MD4: 3f09ff732740edc8ad05c3519bd90c38 MD5: e2c6d562bd35352b73c00a744e9c07c6 SHA-1: f5259423eb42664dec7a32ba6a7cf0d85d13e752
(Ubuntu sha1sum 8.21 && md5sum 8.21) 524f2d1f1356e052d3a4e7cdf190befeac2e0d97 fciv.exe ed976c3b1487c015c3510f3453317eb6 fciv.exe
Different results. But I know the default behavior for sha1sum is to read as text, so let’s use the -b for binary flag to force sha1sum to read it as binary.
sha1sum -b fciv.exe 524f2d1f1356e052d3a4e7cdf190befeac2e0d97 *fciv.exe
Still different from what Windows calculated.
Also, just to make sure Windows is also reading it in binary mode, I wrote a quick Python 3.4 script to generate the hash.
#!/usr/bin/python3 import hashlib f = open('fciv.exe', 'rb') #opens the file read-only and in binary mode m = hashlib.sha1(f.read()) m.hexdigest()
So when I force Python to calculate the hash using binary mode, it returns the same value Windows has been by default. I tried multiple hashing programs for Windows and they all return the same result, so I think it’s safe to assume Windows is using binary encoding.
What if I run the same script on Ubuntu?
Is it the FTP protocol?
Let’s try with a different file. I’ve downloaded a file from my site using Firefox for Windows:
File: HECI_184.108.40.2068.ZIP CRC-32: f516de23 MD4: cdaade9531e3a763df82ab6452376ce4 MD5: d44aba73bfd18b07e19c8531507ddadb SHA-1: 9040c674a3abd052829be04ba415ff55b88cab0a
Then I’ll grab it using Firefox for Ubuntu:
sha1sum HECI_220.127.116.118.ZIP 6d6fa7fa8c0da7b49cffd1f6fae4ce72f231efe8 HECI_18.104.22.1688.ZIP
Is it Ubuntu?
Let’s try the same exercise across two Ubuntu installs. One instance is running 14.04, the other is still on 12.04.5.
14.04 (sha1sum v. 8.21)
[email protected]:/tmp$ sha1sum -b hashthis.txt 6e84d21611dbef9a7898e13bdcc0229c434838ac *hashthis.txt [email protected]:/tmp$ sha1sum hashthis.txt 6e84d21611dbef9a7898e13bdcc0229c434838ac hashthis.txt
12.04.5 (sha1sum v. 8.13)
[email protected]:~$ sha1sum hashthis.txt 6e84d21611dbef9a7898e13bdcc0229c434838ac hashthis.txt [email protected]machine2:~$ sha1sum -b hashthis.txt 6e84d21611dbef9a7898e13bdcc0229c434838ac *hashthis.txt
At least it’s consistent across Linux platforms.
I give up for now.
At this point I’m not sure what’s going on here, but I’ll have to figure out a more reliable way of getting this job done. I might have to cut Windows out of the picture if it can’t return a consistent hash across platforms.
My understanding is that hashes are supposed to be the same given the same input regardless of platform so this really has me stumped. Perhaps ACL permissions are getting factored in to the hash? Those are certainly different.