Hangfire on Cosmos DB and Azure - suddenly start send many requests

I have Hangfire deployed on Azure and using Cosmos DB. It worked correctly. But suddenly start send many requests to database. Number of requests has increased significantly as you can see on screen. And it still is the same. I don't know what is the reason. There was no deploy in this time.

When I call this query in Cosmos DB:

SELECT * FROM c where TimestampToDateTime(c._ts * 1000) > '2021-04-08T22:00:00'

I see only three items, sometimes, four and sometimes six:

[
    {
        "server_id": "edd-deployment-66996d6756-dcxzr:1:938b65a7-196d-4d30-adc5-90101c2781a9",
        "workers": 25,
        "queues": [
            "default"
        ],
        "created_on": 1617290983,
        "last_heartbeat": 1617925294,
        "type": 1,
        "id": "60F6F9F583AC75F979484394ED71F878",
        "_rid": "QO5WANzKklgpmtIBAAAAAA==",
        "_self": "dbs/QO5WAA==/colls/QO5WANzKklg=/docs/QO5WANzKklgpmtIBAAAAAA==/",
        "_etag": "\"3b00c703-0000-0100-0000-606f94ae0000\"",
        "_attachments": "attachments/",
        "_ts": 1617925294
    },

These tasks are heartbeat and they are update all time. But it was in previous time too.

Solution

You need to check what requests are being made to the database to find out what is causing the problem.

Possible problems are:

Recurring jobs executing in parralell
Job failing and generating several retry events