Search code examples
logstashlogstash-configuration

Logstash remove fields by regex


I'm using a FIX filter plugin to process some of our FIX logs. In those messages we receive multiple custom fields. This is outside of the grok filter. I pass the message I care about into this secondary fix plugin

Some of our messages for example look like this:

  "unknown_fields" => [
    [0] "5000",
    [1] "9723",
  ],
  "5000" => "FOOBARVAL",
  "9723" => "BAZBOOHUM",
  "IDSource" => "RIC_CODE",

Question

Is there a way that I can remove tags with mutate or some other filter based on a regular expression(^\d+$)?

More specifically, is there a way that I can remove all of the integer fields that I know will be custom FIX fields (eg. 5000)?


Solution

  • I appreciate the other answer, but I ended up using the prune filter plugin.

    prune {
        blacklist_names => ["[0-9]+", "unknown_fields", "tags"]
    }