Log parsing alert in stackdriver from textPayload?
I'm little bit stuck with alerting in stackdriver. My goal is to make an alert from the parsing log from MySQL error. I have textPayload: "2020-06-24T17:25:05.766410Z 332837 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1073741816 bytes)" messages and i need do alert, when i have 5 messages like that. But i'm stuck with metric, when i'm trying to create and setup i got errors with reg exp. Maybe i need other way to configure that?
Go to query editor and try:
resource.type="gce_instance"
textPayload.message=~"/usr/sbin/mysqld: Out of memory"
This will match all the log entries and then you can create a log-based metric that will count a log entries and then an alert you want (like on the below picture);
When creating an alerting policy make sure that field agregator is set to sum and period is set to whatever period you need. If you set it to 10 minutes if there will be just 4 errors alert won't be triggered. Keep that in mind. Uless they are generated in a very short periods of time but it's something that's specific to your case.
Configuration part should looks like this in my case:
On the other pages select alerting channel etc (pretty clear) and save the alert.
After completion your alert should look like this (I've created a metric & alert specifically for this exaple):
I've triggered an alert on purpose and got an email after about 2 minutes (which is typical) after the incident.