Search code examples
pythonarraysloopsmachine-learningbatch-processing

Batch Chop issue with generator iterator


I am doing the test in coursena and found problem of looping generator with chopping batches:

def batch_generator(items, batch_size):
    
    for i in range(0, len(list(items)), batch_size):
        yield list[i:i+batch_size]

# Test batch generator
def _test_items_generator():
    for i in range(10):
        yield i
        
print(i)
grader.set_answer("a4FK1", list(map(lambda x: len(x), batch_generator(_test_items_generator(), 3))))

The error look like:

TypeError                                 Traceback (most recent call last)
<ipython-input-85-a91baa3cf6fa> in <module>()
      6 
      7 print(i)
----> 8 grader.set_answer("a4FK1", list(map(lambda x: len(x), batch_generator(_test_items_generator(), 3))))

<ipython-input-84-4e82a37b7646> in batch_generator(items, batch_size)
     12     """
     13     for i in range(0, len(list(items)), batch_size):
---> 14         yield list[i:i+batch_size]
     15 
     16     ### YOUR CODE HERE

TypeError: 'type' object is not subscriptable

I dont know where I should fix my problem.


Solution

  • You're trying to subscript the inbuilt list object. You need to convert your input (items) into a list and subscript that:

    def batch_generator(items, batch_size):
        l = list(items)
        for i in range(0, len(l), batch_size):
            yield l[i:i+batch_size]
    

    You can also loop over the items input if you don't want to convert to a list (for example when items is itself a generator):

    def batch_generator(items, batch_size):
        res = []
        for item in items:
            res.append(item)
            if len(res) == batch_size:
                yield(res)
                res = []
        if len(res) > 0:
            yield(res)
    

    Note that we need to check at the end of the loop as to whether there is any leftover data in res and if so, yield that too.