I have Suricata setup as HIDS on a couple of lab instances, and wrote some sample rules to alert on custom User-Headers and internal IPs I can easily trigger for purpose of teaching someone how to use Suricata.
For an advanced use case, I want to output the EVE JSON file somewhere downstream for eventual data analytics and BI use cases.
For that purpose, I want to drop the "noise" from EVE, or have a way for the fast.log to be output in JSON.
For instance, this is what I would consider "noise" as I want to just see triggered
,"event_type":"stats","stats":{"uptime":168,"capture":{"kernel_packets":313,"kernel_drops":0,"errors":0},"decoder":{"pkts":313,"bytes":68519,"invalid":0,"ipv4":305,"ipv6":0,"ethernet":313,"r$
{"timestamp":"2019-08-13T14:29:09.058698+0000","event_type":"stats","stats":{"uptime":176,"capture":{"kernel_packets":313,"kernel_drops":0,"errors":0},"decoder":{"pkts":313,"bytes":68519,"invalid":0,"ipv4":305,"ipv6":0,"ethernet":313,"r$
{"timestamp":"2019-08-13T14:29:17.059944+0000","event_type":"stats","stats":{"uptime":184,"capture":{"kernel_packets":313,"kernel_drops":0,"errors":0},"decoder":{"pkts":313,"bytes":68519,"invalid":0,"ipv4":305,"ipv6":0,"ethernet":313,"r$
I would only want to see stuff like this from fast.log
[**] [1:200002:6] ET USER_AGENTS Suspicious User Agent (BlackSun) [**] [Classification: A Network Trojan was detected] [Priority: 1] {TCP}
So is there a way to get only the Alerts in EVE, or a way to transform Fast.log into JSON?
Found an answer for myself again.
On Line 60 in the YAML, there is a value you can set to "No" for stats - that will eliminate probably 80% of the noise you have. You can go further an eliminate metadata for DNS, TLS, TCP, HTTP, etc. to further reduce your log file if needed.