itertools.groupby in a django template

Question

I'm having an odd problem using itertools.groupby to group the elements of a queryset. I have a model Resource:

from django.db import models 

TYPE_CHOICES = ( 
    ('event', 'Event Room'),
    ('meet', 'Meeting Room'),
    # etc 
)   

class Resource(models.Model):
    name = models.CharField(max_length=30)
    type = models.CharField(max_length=5, choices=TYPE_CHOICES)
    # other stuff

I have a couple of resources in my sqlite database:

>>> from myapp.models import Resource
>>> r = Resource.objects.all()
>>> len(r)
3
>>> r[0].type
u'event'
>>> r[1].type
u'meet'
>>> r[2].type
u'meet'

So if I group by type, I naturally get two tuples:

>>> from itertools import groupby
>>> g = groupby(r, lambda resource: resource.type)
>>> for type, resources in g:
...   print type
...   for resource in resources:
...     print '\t%s' % resource
event
    resourcex
meet
    resourcey
    resourcez

Now I have the same logic in my view:

class DayView(DayArchiveView):
    def get_context_data(self, *args, **kwargs):
        context = super(DayView, self).get_context_data(*args, **kwargs)
        types = dict(TYPE_CHOICES)
        context['resource_list'] = groupby(Resource.objects.all(), lambda r: types[r.type])
        return context

But when I iterate over this in my template, some resources are missing:

<select multiple="multiple" name="resources">
{% for type, resources in resource_list %}
    <option disabled="disabled">{{ type }}</option>
    {% for resource in resources %}
        <option value="{{ resource.id }}">{{ resource.name }}</option>
    {% endfor %}
{% endfor %}
</select>

This renders as:

I'm thinking somehow the subiterators are being iterated over already, but I'm not sure how this could happen.

(Using python 2.7.1, Django 1.3).

(EDIT: If anyone reads this, I'd recommend using the built-in regroup template tag instead of using groupby.)

Answer 1

I think that you're right. I don't understand why, but it looks to me like your groupby iterator is being pre-iterated. It's easier to explain with code:

>>> even_odd_key = lambda x: x % 2
>>> evens_odds = sorted(range(10), key=even_odd_key)
>>> evens_odds_grouped = itertools.groupby(evens_odds, key=even_odd_key)
>>> [(k, list(g)) for k, g in evens_odds_grouped]
[(0, [0, 2, 4, 6, 8]), (1, [1, 3, 5, 7, 9])]

So far, so good. But what happens when we try to store the contents of the iterator in a list?

>>> evens_odds_grouped = itertools.groupby(evens_odds, key=even_odd_key)
>>> groups = [(k, g) for k, g in evens_odds_grouped]
>>> groups
[(0, <itertools._grouper object at 0x1004d7110>), (1, <itertools._grouper object at 0x1004ccbd0>)]

Surely we've just cached the results, and the iterators are still good. Right? Wrong.

>>> [(k, list(g)) for k, g in groups]
[(0, []), (1, [9])]

In the process of acquiring the keys, the groups are also iterated over. So we've really just cached the keys and thrown the groups away, save the very last item.

I don't know how django handles iterators, but based on this, my hunch is that it caches them as lists internally. You could at least partially confirm this intuition by doing the above, but with more resources. If the only resource displayed is the last one, then you are almost certainly having the above problem somewhere.