How do I use Event Store DB client without continued memory usage growth?

Question

I am using the event store client for .Net and I am struggling to find the correct way to use the client. When I register the client as a singleton in the .Net dependency injection and run my application over an extended period of time memory usage grows continuously with each subscription.

I create and register the client in the following way. A full minimal application that experiences the problem can be found here.

var esdbConnectionString = configuration.GetValue("ESDB_CONNECTION_STRING", "esdb://admin:changeit@localhost:2113?tls=false");
var eventStoreClientSettings = EventStoreClientSettings.Create(esdbConnectionString);
var eventStoreClient = new EventStoreClient(eventStoreClientSettings);
services.AddSingleton(eventStoreClient);

My application has a high number of short streams over an extended period of time

To Reproduce

Steps to reproduce the behavior:

1. Register EventStoreClient as singleton as reccomended in the documentation.
2. Subscribe to a very high number of streams over an extended time.
3. Cancel the CancellationToken sent into the stream subscription and let it be garbage collected.
4. Watch memory usage of service grow.

How I am creating and subscribing to streams:

var streamName = CreateStreamName();
var payload = new PingEvent { StreamNr = _currentStreamNumber };
var eventData = new EventData(Uuid.NewUuid(), typeof(PingEvent).Name, EventSerialization.SerializeEventData(payload));
await _client.AppendToStreamAsync(streamName, StreamState.Any, new[] { eventData });

var streamCancellationTokenSource = new CancellationTokenSource(TimeSpan.FromMinutes(30));

await _client.SubscribeToStreamAsync(streamName, FromStream.Start, async (sub, evnt, token) =>
{
    if (evnt.Event.EventType == "PongEvent")
    {
        _previousStreamIsDone = true;
        streamCancellationTokenSource.Cancel();
    }
},
cancellationToken: streamCancellationTokenSource.Token);

Approaches attempted

Registering as Transient or Scoped If I register the client as Transient or Scoped in .Net DI it is throwing thousands of exceptions internally and causing multiple problems.

Manually handling lifetime of client By having a singleton service that handles the lifetime of the client I have attempted to every once in a while dispose of the client and create a new one, ensuring that there exists only one instance of the client at the same time. This results in same problem as registering the service as Transient or Scoped.

I am using version 22.0.0 of the Event Store client in .Net 6 against Event Store Database 21.10.0. The problems happens both when running on windows and on the standard aspnet:6.0 linux docker container.

By inspecting the results of these dotnet-dumps the memory growth seem to be happening inside this HashSet of ActiveCalls in the gRPC client.

I am hoping to find a way of using the client that does not lead to memory growth.

Answer 1

In your reproduction the leaked calls are coming from the extra read that you are issuing while processing an event received on the subscription.

There is an open issue (https://github.com/EventStore/EventStore-Client-Dotnet/issues/219) at the moment to deal with this better, but currently if you issue a read but don't consume all the events and don't cancel the read, then the call remains open. In your case this is happening if the slave has managed to reply Pong before the master has issued the read that results from receiving its own Ping in the subscription. That read will then contain the Ping and the Pong, only the Ping is read, and the call remains open.

For now, if you cancel those reads by passing the cancellation token that you are cancelling into the ReadStreamAsync call in ReadFromStartOfStreamToEnd, it should resolve your problem.

In case it's helpful for you, you can see the number of Current Calls live rather than waiting a long time to see the effect on memory:

dotnet-counters monitor --counters "Grpc.Net.Client" -p <processid>