Why long-lived grpc streaming restarts slowly in a loop

Question

I'm trying to implement an idea described in this blog post.

I'm using long-lived grpc streaming for sending notifications from a backend (server) to applications. A client subscribes for particular notifications and then the server sends them back to the client. When some backend event occurs, clients receive an immediate notification on that event. This rpc will be active while the client is running, because Finish() is never called on a backend side. The backend and all clients are running on a same machine, so network latency is not an issue.

To implement the ability to run the client when the backend is not yet running, I'm trying to subscribe the client every 0.5 seconds in a loop:

std::string server_addr = "unix-abstract:test";
auto client = MakeClient(server_addr);

while (!TryToSubscribeNotifications(client));

If subscription was successful then break; If not, try again.

bool
TryToSubscribeNotifications(Client &client)
{
using std::chrono_literals::operator""ms;

std::thread(&Client::SubscribeNotifications, &client).detach();
std::this_thread::sleep_for(500ms);

return client.IsSubscribed();
}

Inside Client I'm using atomic_bool to set and check subscription status. IsSubscribed() returns this status.

The question is:
Why doesn't the client subscribe for notifications immediately when the backend starts, but only after some random time (usually up to 30 sec) if the client was started about one minute before backend?

Steps to reproduce:

1. run only client, but not backend;
2. wait for one minute. The client should repeatedly print "Notification rpc failed. Attempt #" every 0.5 seconds;
3. run backend;
4. the client should still fail to establish a connection for some random time (usually up to 30 sec).

I'm using grpc callback-based asynchronous API.

Full repo with code example and build instructions: https://github.com/illukin/grpc-longlive-example

Fedora 36
gcc 12.2.1
protobuf 3.21.6.0
gRPC 1.49.1

Answer 1

try this:

List<ChannelOption> channelOptions = new List<ChannelOption>
{
    new ChannelOption("grpc.initial_reconnect_backoff_ms", 1000),
    new ChannelOption("grpc.max_reconnect_backoff_ms", 1000)
};

channel = new Channel(host, port, ChannelCredentials.Insecure, channelOptions);
client = new Commands.CommandsClient(channel);