I'm sorry for asking a bit outlier topic question.
Currently, I'm trying to understand log parser processing and log analysis for anomaly detection in the ML/DL class. There are so many references for this (DeepLog, LogBERT, Loglizer, and so on..)
But one thing I cannot erase one thing in my head is " What is the actual purpose of real-time anomaly detection based on system log?"
According to my understanding, anomaly detection basically gives to support management ( anomaly avoiding or suggesting the anomaly reason and detailed info). and brief workflow is, the model trained the log which log sequence is normal behavior or not. and using that model we can judge whether real-time incoming logs will be an anomaly.
But basically, the system log is printed after something happens. It means real-time anomaly detection seems no need to avoid anomalies because there is no chance to do something in a time gap between the system log and the model doing some action. If this is correct, then we cannot avoid the error or warning and also we don't have enough time to solve issues I think.
As far as I know, Industry (IBM, other cloud companies) uses this kind of anomaly detection system in order to manage the system. But I'm wondering what is the exact role of real-time anomaly detection based on the system log.
Real-time or streaming analysis, also called online anomaly detection systems, detects anomalies in real time as events happen and are recorded. Most of the current methods for real-time analysis use supervised learning. These methods learn patterns offline from normal or labeled logs, logs already known as normal or abnormal, and are then used in production to detect abnormal events in real time.
Log-based anomaly detection faces significant obstacles with supervised methods due to the growing complexity of computer systems, which makes them more vulnerable to failures. Also, getting labeled logs can be costly and time-consuming, as it requires domain experts. So, a streaming anomaly detection system that doesn't rely on normal logs becomes necessary and there is ongoing research to find and develop more unsupervised anomaly detection systems that can analyze data as quickly as it is generated.