Django N+1 problem with serpy.MethodField

Question

I use nplusone to detect N+1 queries.

I have a serpy serializer that serializes Order instances. An Order has a cart consisting of OrderComponent instances, as shown below. Code simplified:

class Order(models.Model):
    objects = OrderManager()
    listings = models.ManyToManyField(to=Listing, through="OrderComponent", related_name="orders")


class OrderComponent(models.Model):
    listing = models.ForeignKey(to=Listing, on_delete=models.PROTECT, related_name="order_components")
    order = models.ForeignKey(to=Order, on_delete=models.PROTECT, related_name="cart")
    nb_units = models.PositiveIntegerField()

A list of serialized Order instances is gotten through OrderListView:

class OrderListView(SimplePaginatedListView):  # Custom base class
    model = Order
    serializer_class = OrderSerializer
    deserializer_class = OrderDeserializer

    def get_queryset(self):
        return super().get_queryset().select_related(
            "fulfillment",  # Used for order.is_fulfilled()
            "cancellation",  # Used for order.is_cancelled()
        ).prefetch_related(
            "cart",  # I don't believe the first two are necessary, but added for testing purposes
            "cart__listing",
            "cart__listing__product",
            "cart__listing__service",
        )

    @http_get(required_permissions=ShopRolePermission.MANAGE_ORDERS)
    def get(self, *args, **kwargs):
        return super().get(*args, **kwargs)

    @http_post(required_permissions=ShopRolePermission.MANAGE_ORDERS)
    def post(self, *args, **kwargs):
        return super().post(*args, **kwargs)

OrderSerializer is defined below. nplusone does not say where an N+1 query was detected, so I have commented out every possible culprit and found the true culprits. I have indicated in comments where they are.

class OrderSerializer(BaseSerializer):
    class SimpleOrderComponentSerializer(BaseSerializer):
        id = serpy.IntField()
        listing_id = serpy.IntField()
        product_name = serpy.MethodField(required=False)  # No N+1
        service_name = serpy.MethodField(required=False)  # No N+1
        nb_units = serpy.IntField()

        def __init__(self, instance=None, many=False, data=None, context=None, **kwargs):
            # Should this be necessary, since I prefetched in get_queryset?
            instance = instance.select_related("listing", "listing__product", "listing__service")
            super().__init__(instance, many, data, context, **kwargs)

        @staticmethod
        def get_product_name(_serializer, obj: OrderComponent):
            # No N+1
            return obj.listing.product.product_name if obj.listing.product else None

        @staticmethod
        def get_service_name(_serializer, obj: OrderComponent):
            # No N+1
            return obj.listing.service.service_name if obj.listing.service else None

    id = serpy.IntField()
    cancelled = serpy.BoolField(attr="is_cancelled")  # Checks if Cancellation instance exists, no N+1
    fulfilled = serpy.BoolField(attr="is_fulfilled")  # Checks if Fulfillment instance exists, no N+1
    cart_products = serpy.MethodField(required=False)  # N+1 !!!
    cart_services = serpy.MethodField(required=False)  # N+1 !!!

    def get_cart_products(self, obj: Order):
        # N+1 detected here.
        return self.SimpleOrderComponentSerializer(obj.cart.filter(listing__product__isnull=False), many=True).data

    def get_cart_services(self, obj: Order):
        # N+1 detected here. Curiously, the number of N+1 queries detected differs between the two.
        return self.SimpleOrderComponentSerializer(obj.cart.filter(listing__service__isnull=False), many=True).data

I cannot for the life of me figure out why my prefetch does not work here. Django Debug Toolbar confirms that it is not a false positive:

SELECT ••• FROM "shop_ordercomponent" INNER JOIN "shop_listing" ON ("shop_ordercomponent"."listing_id" = "shop_listing"."id") INNER JOIN "shop_product" ON ("shop_listing"."product_id" = "shop_product"."id") LEFT OUTER JOIN "shop_service" ON ("shop_listing"."service_id" = "shop_service"."id") WHERE ("shop_ordercomponent"."order_id" = 2 AND "shop_listing"."product_id" IS NOT NULL)
  25 similar queries. 

If I inspect obj in get_cart_services or get_cart_products, I see that obj._prefetched_objects_cache["cart"] is a QuerySet of OrderComponent. If I inspect obj in SimpleOrderComponentSerializer.get_product_name, for example, I see nothing in obj._prefetched_objects_cache["cart"], which I don't expect, since listing.product and listing.service are select_related-ed, not prefetch_related-ed.

I admit I don't fully understand how this works, but I assume select_related populates one-to-one relationships greedily in one query rather than lazily waiting for the related object to be requested.

My initial prefetch_related does not "carry over" into the inner serializer's MethodField handler by the looks of it. nplusone logs:

Potential n+1 query detected on `Order.cart`
Potential n+1 query detected on `Order.cart`
Potential n+1 query detected on `Order.cart`
...  # Repeated many times. Multiple warnings printed for each object.

Is it because cart is a "reverse" relationship? Any help understanding why this isn't working would be greatly appreciated.

Answer 1

I solved the problem by annotating the required information in get_queryset and getting that instead. That way, the prefetched information will be stored in each instance.