How to combine more (at least three) metrics and count result

Question

I need to combine three existing metrics (kube_pod_labels, kube_namespace_labels and kube_pod_status_phase). The fields I need to use are:

• kube_namespace_labels: namespace, team
• kube_pod_labels: pod, namespace, application, team (have to be replaced by kube_namespace_labels.team value)
• kube_pod_status_phase: pod, namespace, phase

I need to get running pods (kube_pod_status_phase) for given application (kube_pod_labels) and count by namespace's team (kube_namespace_labels).

The query which is in my mind is like this: count (kube_pod_labels{application=~"..."} * on(pod) group_left(team) (kube_pod_status_phase{phase="Running"} * on(namespace) group_left(team) kube_namespace_labels{team=~"..."})) by (team), where application and team are multiselects (well also phase can be, but it's not necessary). But this one doesn't work. It return values, but changing phase still returns same counts, which is not correct.

• the Grafana version is 9.30;
• the problem is that the query kube_pod_labels{application=~"..."} * on(pod) group_left(team) (kube_pod_status_phase{phase="Running"} * on(namespace) group_left(team) kube_namespace_labels{team=~"..."}) returns wrong results.

Answer 1

This works as expected:

sum by (team) (
    kube_pod_labels{application=~"..."}
    * on(pod, namespace) group_left(team)
    (
        kube_pod_status_phase{phase="Running"}
            * on(namespace) group_left(team)
            kube_namespace_labels{team=~"..."}
    )
)

Have to use sum instead of count and kube_namespace_labels have to use only namespace.