Search code examples
pythonsqldjangodjango-q

Performance impact of or-ing Q objects in django query


I am performing a query that is or-ing a bunch Q's together and it seems to be taking a lot of time. Here is some psuedo code

query_params = []
for i in range(80): #there are about 80ish Q objects being created
    query_params.append(Q(filter_stuff))

Then I or them all together

query_params = reduce(or_, query_params)

And when I execute the query

query = list(MyModel.objects.filter(query_params))

It hangs for a LONG time. I know this is a pretty general question and it's hard to given a diagnostic without an intimate understanding of the data structure (which would be difficult to give here). But I'm just curious if there is an inherent performance impact of or-ing Q objects in a django query


Solution

  • Was able to shrink down the length of the query significantly by reducing the number of of Qobjects. They were all of the format like:

    q1 = Q(field1=field1_val1, field2=field2_val1)
    q2 = Q(field1=field1_val2, field2=field2_val2)
    #...etc
    

    I ended up grouping them by field1 values:

    q_dict = {field1_val1: [all field_2 values paired with field1_val1], ...}
    

    Then my q objects looked like this:

    for field1_val, field2_vals = q_dict.items():
        query_params.append(Q(field1=field1_val, field2__in=field2_vals))
    

    This ultimately shrank down my number of Q objects significantly and the query ran much faster