Hashing file in Python 3?

Question

In Python 2, one could hash a string by just running:

someText = "a"
hashlib.sha256(someText).hexdigest()

But in Python 3, it needs to be encoded:

someText = "a".encode("ascii")
hashlib.sha256(someText).hexdigest()

But when I try this with a file:

f = open(fin, "r")
sha = hashlib.sha256()
while True:
    data = f.read(2 ** 20).encode("ascii")
    if not data:
        break
    sha.update(data)
f.close()

I get this on many files:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 8: invalid continuation byte

I assume this is because it's a binary file, which likely can't be converted to ASCII.

How can I encode the file without this problem?

can you not do: sha.update(open(filename, "rb").read) ? and why do you do '2 ** 20'? — ikwyl6
– ikwyl6, Commented Jun 23, 2023 at 23:20

Tim Peters · Accepted Answer · 2013-10-13 05:15:07Z

On Unix systems, in Python 2 there was no distinction between binary- and text-mode files, so it didn't matter how you opened them.

But in Python 3 it matters on every platform. sha256() requires binary input, but you opened the file in text mode. That's why @BrenBam suggested you open the file in binary mode.

Since you opened the file in text mode, Python 3 believes it needs to decode the bits in the file to turn the bytes into Unicode strings. But you don't want decoding at all, right?

Then open the file in binary mode, and you'll read byte strings instead, which is what sha256() wants.

By the way, your:

someText = "a".encode("ascii")
hashlib.sha256(someText).hexdigest()

can be done more easily in a related way:

hashlib.sha256(b"a").hexdigest()

That is, pass it the binary data directly, instead of bothering with encoding a Unicode string (which the literal "a" is).

BrenBarn · Accepted Answer · 2013-10-13 05:17:26Z

6

Try opening the file in binary mode with open(fin, "rb").

answered Oct 13, 2013 at 5:17

BrenBarn

253k39 gold badges421 silver badges392 bronze badges

Comments

Stephen Rauch · Accepted Answer · 2018-02-12 01:19:21Z

0

I have programmed a module which is able to hash big files with different algorithms.

pip3 install py_essentials

Use the module like this:

from py_essentials import hashing as hs
hash = hs.fileChecksum("path/to/the/file.txt", "sha256")

Take a look at the documentation.

edited Feb 12, 2018 at 1:19

Stephen Rauch♦

50.1k32 gold badges118 silver badges143 bronze badges

answered Feb 12, 2018 at 0:52

1cedsoda

6484 silver badges18 bronze badges

Collectives™ on Stack Overflow

Hashing file in Python 3?

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related