Filtering information in Google Cloud Traces from different gke clusters in same GCP project

Question

I have one GCP project in which there are two GKE clusters, one for development and preproduction workloads and another one for production.

I am using Google Cloud Trace to correlate the information of the different Spring Boot microservice invocations.

The problem is that the Trace list view shows all calls from the two clusters at the same time so it's hard to tell apart production calls from the others:

Is there a way to filter calls from different clusters?

If not, one solution would be to isolate the GKE production cluster in its own GCP project. Is this a good practice?

Answer 1

To solve this I had to customize the span information in the Spring Boot applications, adding the namespace:

@Component
class SpanCustomizerFilter(private val spanCustomizer: SpanCustomizer) : WebFilter {

    override fun filter(exchange: ServerWebExchange, chain: WebFilterChain): Mono<Void> {
        spanCustomizer.tag("namespace", System.getenv("POD_NAMESPACE"))
        return chain.filter(exchange)
    }
}

Then in the Add filter section I could add namespace:mynamespace-dev and get just the traces I needed: