Efficiently determining if large sorted numpy array has only unique values

Question

I have a very large numpy array and I want to sort it and test if it is unique.

I'm aware of the function numpy.unique but it sorts the array another time to achieve it.

The reason I need the array sorted a priori is because the returned keys from the argsort function will be used to reorder another array.

I'm looking for a way to do both (argsort and unique test) without the need to sort the array again.

Example code:

import numpy as np
import numpy.random

# generating random arrays with 2 ^ 27 columns (it can grow even bigger!)
slices = np.random.random_integers(2 ** 32, size = 2 ** 27)
values = np.random.random_integers(2 ** 32, size = 2 ** 27)

# get an array of keys to sort slices AND values
# this operation takes a long time
sorted_slices = slices.argsort()

# sort both arrays
# it would be nice to make this operation in place
slices = slices[sorted_slices]
values = values[sorted_slices]

# test 'uniqueness'
# here, the np.unique function sorts the array again
if slices.shape[0] == np.unique(slices).shape[0]:
    print('it is unique!')
else:
    print('not unique!')

Both the arrays slices and values have 1 row and the same (huge) number of columns.

Thanks in advance.

Answer 1

You can check whether there are two or more equal values next to each other (non-unique values in a sorted array) by comparing their difference to 0

numpy.any(numpy.diff(slices) == 0)

Be aware though that numpy will create two intermediate arrays: one with the difference values, one with boolean values.