Django channels "ERROR Y of N channels over capacity in group subscriptions"

Question

I'm doing load testing with my Django app providing GraphQL Subscriptions using Django channels and a redis Channels layer (django, graphene-django, channels, graphene-subscriptions, channels-redis). As ASGI server I'm using daphne right now. I use nginx as proxy. The periodicity with which the backend publishes messages via GraphQL Subscriptions depends on the periodicity of messages the backend receives via MQTT. I'm increasing the periodicity with which an external data provider publishes messages to the MQTT broker, means the periodicity with which the backend has to process these messages and publish messages via GraphQL Subscriptions. I'm facing the following error:

2020-03-11 08:33:58,464 ERROR    2 of 12 channels over capacity in group subscriptions

It seems like this issue is caused by channels_redis. Can I scale the infrastructure to workaround this issue?

Answer 1

The default capacity is 100 messages and the default message expiration time is 60 seconds. So if the the channel is never read within these capacity / time constraints, it will fill up.

One reason why a channel might fill up is when the connection is never properly closed. In this case the channel will remain in the group and eventually fill up.

One way to mitigate this is to have enough capacity and a short timeout. You can alter the configuration in the Django settings in the following way:

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": REDIS_URL,  # or where your redis server lives
            "capacity": 1500,  # default 100
            "expiry": 10,  # default 60
        }
     }
}