google spanner singleton reconnect on fault

Question

I am creating a SpannerSingleton to stay connected for the duration of the app's life. I'm interested in connection durability... if there is a session/connection issue how can I recreate a session?

One idea was to spawn a new connection increasing the setMaxSessions to a higher number if more than 90% of the pool is exhausted. Like the opposite of exponential backoff? But where / how can I do that? I could not find anything in the client library that would let me monitor the pool status or client count.

I went with bill-pugh-singleton cuz it seemed like a good choice...

Here is what I have:

public class SpannerSingleton {

    private static Spanner spanner;
    private static SpannerOptions options;

    private static SessionPoolOptions sessionPoolOps = SessionPoolOptions
            .newBuilder()
            .setMaxSessions(1000)  // 1000 concurrent queries
            .setMinSessions(100)  // keep 100 alive
            .setMaxIdleSessions(100)  // how many to keep from being idle and closed
            .build(); 

    private SpannerSingleton() {
        try {
            options = SpannerOptions
                    .newBuilder()
                    .setSessionPoolOption(sessionPoolOps)
                    .build();

            spanner = options.getService();

        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static class SingletonHelper{
        private static final Spanner CONNECTION = new SpannerSingleton().spanner;
    } 

    public static synchronized Spanner getSpanner() {
        return SingletonHelper.CONNECTION;
    }

}

I use a Factory pattern to make the dbClient

public SpannerFactory {

    private static Spanner spanner = SpannerSingleton.getSpanner();
    private static DatabaseId dbId;

    public static DatabaseClient getConnection(String instance) {
        if (Util.isEmpty(instance)) return null;

        if ("mickey".equalsIgnoreCase(instance)) {
            dbId = DatabaseId.of(spanner.getOptions().getProjectId(), "instance1", "mickey");
        }

        if ("mouse".equalsIgnoreCase(instance)) {
            dbId = DatabaseId.of(spanner.getOptions().getProjectId(), "instance1", "mouse");
        }

        return spanner.getDatabaseClient(dbId);

    }
}

What I would like to add is something that would check for the connection pool to see how close to starved we are and then recreate itself... I might be over thinking it, but what might happen if the connection is disrupted?

Answer 1

Client library should take care of maintaining healthy session pool, user shouldn't have to worry about sessions/connections explicitly.

As documented in java client, if you set MaxSessions correctly - client will take care of maintaining those many sessions.

At a high level, the flow would be like:

If currentSessions < MaxSessions {
   if !idleSessions.empty()
        use an idle session.
   else
        CreateNewSession.
} else {
   Block/Fail based on action chosen in : ActionOnExhaustion.
}

If you want to avoid small overhead in CreateSession as part of request processing, one recommended option is to keep minSessions and maxSessions same as your concurrent TPS requirement so that at the beginning we have those many sessions ready to use.

For additional details like session monitoring, keeping idle sessions alive : please refer to documentation at : https://cloud.google.com/spanner/docs/sessions