Search code examples
stackdrivergoogle-cloud-stackdriver

Advanced filter can't express ISNULL?

These two filters return zero results:

  • resource.labels:* AND resource.labels.namespace_name:*
  • resource.labels:* AND NOT resource.labels.namespace_name:*

While this one returns plenty:

  • resource.labels:*

I have three questions about this:

  1. What's going on here?
  2. More importantly, how do I exclude a particular value of namespace_name while not excluding records that don't define namespace_name ?
  3. Similarly, how do I write a filter for all records that don't define namespace_name?
Solution

  • I work on Stackdriver Logging and have worked with the code that handles queries.

    You are correct: something's up with the presence operator (:*), and it works differently than the other operators. As a result the behavior of a negated presence operator is not intuitive (or particularly useful).

    We consider this a bug, and it's something that I'd really like to fix; however, fixing this class of bug is a lengthy process, so I've proposed some workarounds.

    1. What's going on here?

    I cannot reproduce your first "zero result" filter: resource.labels:* AND resource.labels.namespace_name:*

    This gives me a large list of logs that contain the namespace_name label. For what it's worth, resource.labels.namespace_name:* implies resource.labels:*, so really you only need the latter half of this filter.

    Your second "zero result" filter: resource.labels:* AND NOT resource.labels.namespace_name:*

    ... runs into a bug where field presence check (:*) does not interact properly with negation.

    1. More importantly, how do I exclude a particular value of namespace_name while not excluding records that don't define namespace_name ?

    While not required by the logging API, GCP-emitted resources generally emit the same sets of labels for a given resource type. You can take advantage of this by using resource.type to isolate resources-with-label from resources-without-label, then only apply the label constraint to the resources-with-label clause:

    (resource.type != "k8s_container") OR
(resource.type = "k8s_container" AND resource.labels.namespace_name != "my-value")

    Here, we are relying on all k8s_container-type entries having the namespace_name label, which should generally be the case. You can modify this to select multiple Kubernetes-prefixed resources:

    (NOT resource.type:"k8s_") OR
(resource.type:"k8s_" AND resource.labels.namespace_name != "my-value")

    ... or use a complex resource.type clause to specifically select which you want to include/exclude from the namespace matching.

    (NOT (resource.type = "k8s_container" OR resource.type = "k8s_pod")) OR
((resource.type = "k8s_container" OR resource.type = "k8s_pod") AND resource.labels.namespace_name != "my-value")

    You cannot query for a k8s_container type that does not have the namespace_name label, but those should generally not be emitted in the first place.

    1. Similarly, how do I write a filter for all records that don't define namespace_name?

    You can't do this right now because of the bug. I think your best bet is to identify all of the resource types that use namespace_name and exclude those types with a resource.type filter:

    NOT (
  resource.type = "k8s_container" OR
  resource.type = "k8s_pod" OR
  resource.type = "knative_revision")

    Note that, as mentioned earlier, while it's possible (allowed by the API) to have a k8s_container resource without a namespace_name label, emitted k8s_container logs should generally have the label.