(Ubuntu) sha1sum and (Windows) fciv are returning different SHA1 hashes

I’m shuffling a few 1TB images from a Ubuntu machine to a Windows box. Thinking I’d be slick, I used sha1sum to calculate the SHA1 hash of the first image before sending.

sha1sum system-image.vdk

Then I used the File Checksum Integrity Verifier (fciv.exe) by Microsoft to do the same once I had used ncftp to transfer the image to the Windows box.

fciv.exe -sha1 system-image.vdk

But I ran into an immediate problem. The hashes being returned were not the same.

Is it ncftp?

I tried it using regular ftp. Same deal.

Is it an encoding issue?

I’ve been testing with text files, which leaves some room for differing interpretations by the OS. So let’s play with a binary again downloaded via ftp.

(HashCheck for Windows)
File: fciv.exe
CRC-32: 71cbdfd9
MD4: 3f09ff732740edc8ad05c3519bd90c38
MD5: e2c6d562bd35352b73c00a744e9c07c6
SHA-1: f5259423eb42664dec7a32ba6a7cf0d85d13e752
(Ubuntu sha1sum 8.21 && md5sum 8.21)
524f2d1f1356e052d3a4e7cdf190befeac2e0d97  fciv.exe
ed976c3b1487c015c3510f3453317eb6  fciv.exe

Different results. But I know the default behavior for sha1sum is to read as text, so let’s use the -b for binary flag to force sha1sum to read it as binary.

sha1sum -b fciv.exe
524f2d1f1356e052d3a4e7cdf190befeac2e0d97 *fciv.exe

Still different from what Windows calculated.

Also, just to make sure Windows is also reading it in binary mode, I wrote a quick Python 3.4 script to generate the hash.

import hashlib
f = open('fciv.exe', 'rb') #opens the file read-only and in binary mode
m = hashlib.sha1(f.read())

So when I force Python to calculate the hash using binary mode, it returns the same value Windows has been by default. I tried multiple hashing programs for Windows and they all return the same result, so I think it’s safe to assume Windows is using binary encoding.

What if I run the same script on Ubuntu?


Is it the FTP protocol?

Let’s try with a different file. I’ve downloaded a file from my site using Firefox for Windows:

File: HECI_7.0.0.1118.ZIP
CRC-32: f516de23
MD4: cdaade9531e3a763df82ab6452376ce4
MD5: d44aba73bfd18b07e19c8531507ddadb
SHA-1: 9040c674a3abd052829be04ba415ff55b88cab0a

Then I’ll grab it using Firefox for Ubuntu:

sha1sum HECI_7.0.0.1118.ZIP
6d6fa7fa8c0da7b49cffd1f6fae4ce72f231efe8  HECI_7.0.0.1118.ZIP

Still different.

Is it Ubuntu?

Let’s try the same exercise across two Ubuntu installs. One instance is running 14.04, the other is still on 12.04.5.

14.04 (sha1sum v. 8.21)

[email protected]:/tmp$ sha1sum -b hashthis.txt 
6e84d21611dbef9a7898e13bdcc0229c434838ac *hashthis.txt
[email protected]:/tmp$ sha1sum hashthis.txt 
6e84d21611dbef9a7898e13bdcc0229c434838ac  hashthis.txt

12.04.5 (sha1sum v. 8.13)

[email protected]:~$ sha1sum hashthis.txt 
6e84d21611dbef9a7898e13bdcc0229c434838ac  hashthis.txt
[email protected]machine2:~$ sha1sum -b hashthis.txt 
6e84d21611dbef9a7898e13bdcc0229c434838ac *hashthis.txt

At least it’s consistent across Linux platforms.

I give up for now.

At this point I’m not sure what’s going on here, but I’ll have to figure out a more reliable way of getting this job done. I might have to cut Windows out of the picture if it can’t return a consistent hash across platforms.

My understanding is that hashes are supposed to be the same given the same input regardless of platform so this really has me stumped. Perhaps ACL permissions are getting factored in to the hash? Those are certainly different.

Leave a Reply