Search code examples
djangoormdjango-orm

How to get unique data for each column from the database in one query (Django ORM)?


I need to get UNIQUE data from EACH column I have a list of fields:

MY_FIELDS = ['id', 'created', 'And_more_28']

My HORRIBLY INEFFECTIVE code that produces the correct result

qs = Sheets.objects.all()
v = []
v += [list(qs.values(v).distinct()) for v in MY_FIELDS]

I get what I need.... But at what cost? (28 requests is painful) Tell me how to make one query out of this....

This is how it produces non-unique data for EACH column (tried it)

qs = Sheets.objects.all().values(*MY_FIELDS).distinct()

Solution

  • Your second approach does produce unique values, but these are unique combinations. So if there are 10 fields with each 5 values, that already can result in at most 510 results, so it scales dramatically.

    But the good news is, we can solve this, by aggregating all columns into distinct lists. Indeed, with ArrayAgg [Django-doc]:

    MY_FIELDS = ['id', 'created', 'And_more_28']
    
    from django.contrib.postgres.aggregates import ArrayAgg
    
    qs = Sheets.objects.aggregate(
        **{'values_{f}': ArrayAgg('f', distinct=True) for f in MY_FIELDS}
    )
    v = [qs[f'values_{f}'] for f in MY_FIELDS]

    This will produce lists of distinct values for each column, and all in one single query.