I am designing a system with a single kubernetes deployment for multiple tenants, but there are multiple databases, queues etc. per customer. Anything stateless is shared, and anything with state is separate for each tenant. Based on the request host (tenant1.company.com or tenant2.company.com), the code will connect to the corresponding databases and queues.
How should readiness probes be designed in this case, where my pod is designed to work for multiple tenants?
I can think of the following options, none of which seem correct:
It feels like if I have separation at resources level to support multiple tenants (this is a B2B multi-tenant, takes time and effort to onboard new tenant), I also need to have separation at Kubernetes deployment level.
Is this the standard approach - either have complete separation at all levels, or have one single unified system with same shared resources? If not, how do I design the readiness probes?
As per my understanding, you are trying to extend Kubernetes Pod's readiness probe to reflect Application health for specific tenant. Unfortunately, Readiness probe is not designed for that.
The only purpose of Kubernetes readiness probe (even the new feature Pod Ready++) is to reflect the certain Pod ability to serve the traffic. Deployment and StatefulSet controllers take into account Pod readiness state during the rolling-update process.
You can brake the whole update mechanics if you set the readiness probe to depend on external to the Pod components or a network endpoints connectivity. The correct way of using Readiness probe is to check only the Pod's internal components' conditions.
Kubernetes documentation pages:
For certain simple application or microservices that only contains one Pod, it may reflect also the state of the application. But usually, application architecture is much more complex and contains many parts, each of which may have dependencies.
Sometimes it's cheaper and simpler to create your own health check loop in the frontend app (www.example.com/healthz) that reflects the whole App health, taking to account all its components statuses and their dependencies, or collect and aggregate JSON statuses from other components.
In Kubernetes world, components/apps are usually Services that balance the traffic to one or more Pods. Thus, a component is healthy if at least one Pod behind the corresponding Service is in Ready state. The number of Ready Pods behind the Service tells more about App Performance, than about the App Health.
As per my ability to imagine your App design:
This way you can easily scale application parts if some tenant needs more resources.
To manage deployments I would use Helm charts. This way I can easily deploy one more tenant or remove/update the existing one.
There are many different solutions to monitor an Application Health, Performance, collect metrics and logs and take actions if certain conditions are met. It's just a brief list of most popular solutions:
PS: In case you want to implement a circuit breaker for the tenants, Istio has built-in functionality.