I am using Python 3.7.1 for making minhash a list of string. The code is as follows.
import mmh3
import random
import string
import itertools
from datasketch import MinHash
def grouper(iterable,n=2):
return ["".join(x) for x in list(itertools.permutations(iterable, n))]
def _hash_func(d):
return mmh3.hash(d)
def _run_minhash(data, seed):
m = MinHash(num_perm=128 ,hashfunc=_hash_func)
for d in data:
m.update(d.encode('utf8'))
return m.count()
if __name__ == '__main__':
st = string.ascii_uppercase
ngrams = grouper(st,n=4)
print(_run_minhash(ngrams,seed=12))
To get speedup I am using mmh3.hash as mentioned in datasketch.Minhash documentation (mmh3.hash32 not available anymore) which produces the following error
TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
Traceback (most recent call last):
File "hashes.py", line 41, in
print(_run_minhash(ngrams,seed=12))
File "hashes.py", line 35, in _run_minhash
m.update(d.encode('utf8'))
File "/home/nithin/miniconda3/lib/python3.7/site-packages/datasketch/minhash.py", line 134, in update
phv = np.bitwise_and((a * hv + b) % _mersenne_prime, np.uint64(_max_hash))
TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
I did not find a specific reason the reason and solution for this. How should I resolve this?. Or is there any other way to speed up the minhash calculation. Thanks in advance for your time – if I’ve missed out anything, over- or under-emphasized a specific point let me know in the comments.
same problem here and problem solved by adding signed=False to the call to the mmh3.hash function
def _hash_func(b):
return mmh3.hash(b,signed=False)