How can I setup network traffic alerts on a Linux machine using Prometheus?

Question

I am using Prometheus to monitor network traffic on Linux machines. I see several useful metrics like node_network_receive_bytes_total, node_network_transmit_bytes_total, and node_network_receive_errs_total. How can I setup alerts for network traffic when I am not aware of the network bandwidth of the machine?

Answer 1

you should use some of queries to have good results of network monitoring. i am using some queries on Grafana and i share them with you:

1. popular queries:

Query-outbound sum (irate(node_network_transmit_bytes{hostname=~"$hostname", device!~"lo|bond[0-9]|cbr[0-9]|veth.*"}[1m])) by (hostname) > 0

Legend format: {{hostname}} - {{device}} - outbound

Query-inbound sum (irate(node_network_receive_bytes{hostname=~"$hostname", device!~"lo|bond[0-9]|cbr[0-9]|veth.*"}[1m])) by (hostname) > 0

Legend format: {{hostname}} - {{device}} - inbound

2. complicated queries:

network terafic of eno(or any things you want) devices:

Legend format: {{hostname}} - ({{device}})_in

irate(node_network_receive_bytes{hostname=~'$hostname',device=~"^en.*"}[5m])*8

Legend format: {{hostname}} - ({{device}})_out

irate(node_network_transmit_bytes{hostname=~'$hostname',device=~"^en.*"}[5m])*8

netstas:

Legend format: {{hostname}} established

node_netstat_Tcp_CurrEstab{hostname=~'$hostname'}

udp stat:

irate(node_netstat_Udp_InDatagrams{hostname=~"$hostname"}[5m])

irate(node_netstat_Udp_InErrors{hostname=~"$hostname"}[5m])

irate(node_netstat_Udp_OutDatagrams{hostname=~"$hostname"}[5m])

irate(node_netstat_Udp_NoPorts{hostname=~"$hostname"}[5m])

contract

Legend format: Queue Used ({{hostname}})

node_nf_conntrack_entries{hostname=~"$hostname"}/node_nf_conntrack_entries_limit{hostname=~"$hostname"}

pleas attention to hostname . it is a template variable on Grafan . and legend format is a label parsing for metrics on Grafana.