Anomaly Detection
Models, forecasts, anomalies, and false-positives.
I am investigating anomaly detection techniques for machine sensor data. This site will hold notes, observations, and results as I proceed.
Anomaly detection is used to detect events that significantly deviate from a usual pattern. This is extremely important for monitoring systems. These days, systems are becoming increasingly complex with deep hierarchies of devices and large arrays of nodes. We’re interested in being informed about anomalous activity without having to generate models or perform guesswork about what kinds of models fit.
The goal is to be able to detect anomalous changes in computer system time series metrics on-the-fly, efficiently, and at high resolution (one-second granularity). There are possibly thousands of different metrics on each of the kinds of systems I’d like to monitor, and with one-second resolution sampling, this means thousands of new observations every second. With these conditions, false-positives must be minimized, as even low-probability events become quite common at high rates.
All of the content shown will be available on the GitHub repository.