Search code examples
pythonpython-3.xmemoryview

I still don't understand the point of memoryview


I read through the questions and answers or What exactly is the point of memoryview in Python. I still don't see the point.

The example in the answer seems logical at first, but when I construct a third test case, where I scan through the bytesobject by index, it as fast as with the memoryview.

import time


# Scan through a bytes object by slicing
for n in (100000, 200000, 300000, 400000):
    data = b'x' * n
    start = time.time()
    b = data
    while b:
        b = b[1:]
    print('bytes sliced  ', n, time.time() - start)

# Scan through a bytes object with memoryview
for n in (100000, 200000, 300000, 400000):
    data = b'x' * n
    start = time.time()
    b = memoryview(data)
    while b:
        b = b[1:]
    print('memoryview    ', n, time.time() - start)

# Scan through a bytes object by index
for n in (100000, 200000, 300000, 400000):
    data = b'x' * n
    start = time.time()
    b = data
    for i in range(n):
        b = b[i+1:]
    print('bytes indexed ', n, time.time() - start)

Output:

bytes sliced   100000 0.16396498680114746
bytes sliced   200000 0.6180000305175781
bytes sliced   300000 1.541727066040039
bytes sliced   400000 2.8526365756988525
memoryview     100000 0.02300119400024414
memoryview     200000 0.04699897766113281
memoryview     300000 0.0709981918334961
memoryview     400000 0.0950019359588623
bytes indexed  100000 0.027998924255371094
bytes indexed  200000 0.05700063705444336
bytes indexed  300000 0.08800172805786133
bytes indexed  400000 0.1179966926574707

One of the arguments was, that you can simply pass a memoryview object to struct.unpack. But you can absolutely do the same with a bytes objects. In my understanding it boils down to the same as memoryview eventually has to copy the slice, too.

Just sticking with bytes seems so much simpler, if you don't do stupid things.


Solution

  • Your first two benchmarks essentially nibble off a single byte from the left until there is nothing left.

    For the bytes example, this does N copies, for memoryview there is never a copy, just an adjustment of the length of the view

    Your last example isn't at all similar, instead of nibbling off a single byte, you nibble off a progressively larger number of bytes (b[1:] b[2:] b[3:]) -- eventually the string is exhausted and you're slicing an empty string (more precisely when i * (i + 1) / 2 > n). For example with the 100,000 byte sequence, you're doing noops after 446 iterations.