Search code examples
pythondjangosqlitedjango-admindjango-admin-filters

Django very large reverse ForeignKey query


I am using Django admin to filter objects on their reverse ForeignKey objects field value. The App has a ForeignKey to Contact and I am trying to filter Contacts by Apps category in Django admin. The problem is that the query is very large and I am getting a timeout error. There are around 300k Contact objects and around 1M of Apps. There are 500 results per page in Django admin. The db indexes and prefetch_related are added. What else should I do to optimise Django admin ? I am using sqlite db.

The code:

class App(models.Model):
    contact = models.ForeignKey('Contact', related_name='apps', null=True)
    category = models.TextField(blank=True, null=True, db_index=True)
    store = models.IntegerField(choices=STORE_TYPES, db_index=True)

    class Meta:
        index_together = [
            ['category', 'store'],
        ]

# admin:

class ContactAdmin(admin.ModelAdmin):
    list_filter = (
        AppCategory,
    )

    def queryset(self, request):
        return super(ContactAdmin, self).queryset(request).prefetch_related(
            'apps',
            'to_contact',
        )

# the main list_filter that is causing troubles:

class AppCategory(admin.SimpleListFilter):
    title = 'app category'
    parameter_name = 'app_category'

def lookups(self, request, modelAdmin):

    return [
        ('Action', 'Action'),
        ('Adventure', 'Adventure'),
        ('Arcade', 'Arcade'),
        ('Board', 'Board'),
        ('Books', 'Books'),
        ('Books & Reference', 'Books & Reference'),
        ('Business', 'Business'),
        ('Card', 'Card'),
        ('Casino', 'Casino',),
        ('Casual', 'Casual'),
        ('Catalogs', 'Catalogs'),
        ('Comics', 'Comics'),
        ('Communication', 'Communication'),
        ('Education', 'Education'),
        ('Educational', 'Educational'),
        ('Entertainment', 'Entertainment'),
        ('Family', 'Family'),
        ('Finance', 'Finance'),
        ('Food & Drink', 'Food & Drink'),
        ('Games', 'Games'),
        ('Health & Fitness', 'Health & Fitness'),
        ('Libraries & Demo', 'Libraries & Demo'),
        ('Lifestyle', 'Lifestyle'),
        ('Media & Video', 'Media & Video'),
        ('Medical', 'Medical'),
        ('Music', 'Music'),
        ('Music & Audio', 'Music & Audio'),
        ('Navigation', 'Navigation'),
        ('News', 'News'),
        ('News & Magazines', 'News & Magazines'),
        ('Personalization', 'Personalization'),
        ('Photo & Video', 'Photo & Video'),
        ('Photography', 'Photography'),
        ('Productivity', 'Productivity'),
        ('Puzzle', 'Puzzle'),
        ('Racing', 'Racing'),
        ('Reference', 'Reference'),
        ('Role Playing', 'Role Playing'),
        ('Shopping', 'Shopping'),
        ('Simulation', 'Simulation'),
        ('Social', 'Social'),
        ('Social Networking', 'Social Networking'),
        ('Sports', 'Sports'),
        ('Strategy', 'Strategy'),
        ('Tools', 'Tools'),
        ('Transportation', 'Transportation'),
        ('Travel', 'Travel'),
        ('Travel & Local', 'Travel & Local'),
        ('Trivia', 'Trivia'),
        ('Utilities', 'Utilities'),
        ('Weather', 'Weather'),
        ('Word', 'Word')
    ]

def queryset(self, request, queryset):
    if not self.value():
        return queryset
    else:
        qs = queryset.filter(
            apps__category=self.value()
        )
        return qs

Solution

  • I found out the answer. If you are filtering on one reverse ForeignKey relationship (apps__category in this case) everything works fine. But I forgot to mention that I am combining two filters at the right side of Django admin page - the other one is also a reverse ForeignKey - apps__store. When you are doing that, two completely same INNER JOINs are performed and this results in a server timeout. When I am using these filters separately, everything works fine.

    See: https://code.djangoproject.com/ticket/16554