Search code examples
fluentd

Rename subkey in json output

Is it possible to rename a subkey in JSON or add a new subkey?

For example, I have this log output:

{
    "kubernetes": {
        "pod_name": "kube-apiserver-tst",
        "namespace_name": "kube-system",
        "pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
        "labels": {
            "component": "kube-apiserver",
            "tier": "control-plane"
        }
}

I know that with record_transformer I can add a new key:

<record>
  pod_labels "something ..."
</record>

but it seems that it can only create a new key at the root of JSON:

{
    "kubernetes": {
        "pod_name": "kube-apiserver-tst",
        "namespace_name": "kube-system",
        "pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
        "labels": {
            "component": "kube-apiserver",
            "tier": "control-plane"
        }
    "pod_labels": "something ..."
}

But can I make it look like this?

{
    "kubernetes": {
        "pod_name": "kube-apiserver-tst",
        "namespace_name": "kube-system",
        "pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
        "labels": {
            "component": "kube-apiserver",
            "tier": "control-plane"
        }
        "pod_labels": {
            "component": "kube-apiserver",
            "tier": "control-plane"
        }
}

or this:

{
    "kubernetes": {
        "pod_name": "kube-apiserver-tst",
        "namespace_name": "kube-system",
        "pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
        "pod_labels": {
            "component": "kube-apiserver",
            "tier": "control-plane"
        }
}
Solution

  • The JSON is not valid. A closing curly brace is missing.

    Here's the valid JSON:

    {
    "kubernetes": {
        "pod_name": "kube-apiserver-tst",
        "namespace_name": "kube-system",
        "pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
        "labels": {
            "component": "kube-apiserver",
            "tier": "control-plane"
        }
    }
}

    Minified JSON (echo '{JSON}' | jq -c .):

    {"kubernetes":{"pod_name":"kube-apiserver-tst","namespace_name":"kube-system","pod_id":"93a2b43a-46e6-4539-8674-06dede2619fa","labels":{"component":"kube-apiserver","tier":"control-plane"}}}

    The record_transformer filter plugin can be used with Ruby support (via enable_ruby option) to manipulate an existing key and then the unwanted keys can be removed with its remove_keys option.

    Here's the sample config:

    <filter debug.test>
  @type         record_transformer
  enable_ruby   true
  <record>
    temp        ${ l = record["kubernetes"]["labels"]; record["kubernetes"]["pod_labels"] = l; nil; }
  </record>
  remove_keys   temp, $.kubernetes.labels
</filter>

    Here's the complete test:

    fluent.conf

    <source>
  @type       forward
</source>

<filter debug.test>
  @type         record_transformer
  enable_ruby   true
  <record>
    temp        ${ l = record["kubernetes"]["labels"]; record["kubernetes"]["pod_labels"] = l; nil; }
  </record>
  remove_keys   temp, $.kubernetes.labels
</filter>

<match debug.test>
  @type       stdout
</match>

    Start fluentd with this config:

    fluentd -c fluent.conf

    On another terminal, send an event with fluent-cat (echo '{JSON}' | fluent-cat debug.test):

    echo '{"kubernetes":{"pod_name":"kube-apiserver-tst","namespace_name":"kube-system","pod_id":"93a2b43a-46e6-4539-8674-06dede2619fa","labels":{"component":"kube-apiserver","tier":"control-plane"}}}' | fluent-cat debug.test

    In fluentd logs, you should see the desired output:

    2022-02-16 23:08:25.919967225 +0500 debug.test: {"kubernetes":{"pod_name":"kube-apiserver-tst","namespace_name":"kube-system","pod_id":"93a2b43a-46e6-4539-8674-06dede2619fa","pod_labels":{"component":"kube-apiserver","tier":"control-plane"}}}

    Formatted output with jq (echo '{JSON}' | jq .):

    echo '{"kubernetes":{"pod_name":"kube-apiserver-tst","namespace_name":"kube-system","pod_id":"93a2b43a-46e6-4539-8674-06dede2619fa","pod_labels":{"component":"kube-apiserver","tier":"control-plane"}}}' | jq .

    Output:

    {
  "kubernetes": {
    "pod_name": "kube-apiserver-tst",
    "namespace_name": "kube-system",
    "pod_id": "93a2b43a-46e6-4539-8674-06dede2619fa",
    "pod_labels": {
      "component": "kube-apiserver",
      "tier": "control-plane"
    }
  }
}