Search code examples
pythonpython-3.xhashlibfunctools

Hashlib MemoryError in Python 3.5 but not in 2.7

I've been porting a set of Python 2.7 scripts to Python 3.5 so that I can use some libraries that aren't available in 2.7, but I'm getting MemoryError from this code that worked previously:

import hashlib, functools
sha2h = hashlib.sha256()
with open('/path/to/any/file', 'rb') as f:
        [sha2h.update(chunk) for chunk in iter(functools.partial(f.read, 256), '')]

As far as I can tell, this is the proper way to get a SHA256 hash of a file. I can't seem to find anything about this issue. If it helps, here's the traceback when the above code is run from the shell:

File "<stdin>", line 2, in <module>
File "<stdin>", line 2, in <listcomp>

Also tried replacing

sha2h = hashlib.sha256()

with

sha2h = hashlib.new('sha256')

to match the documentation for hashlib, but this yielded no change in outcome.

Any help or insight would be greatly appreciated!

Solution

  • On Python 3, you need to set b'' instead of '' as the sentinel value for the iter call:

    iter(functools.partial(f.read, 256), b'')

    You also really shouldn't be using a list comprehension for side effects like this, but if you're porting existing code that does that, it's probably best to just leave that for now.