Grafana has released a tool that detects outliers as part of Grafana’s machine learning toolkit. Outlier detection can be used to monitor a group of similar things and alert when some of them start behaving differently from the norm.

What does outlier detection do?
Modern applications deployed and scaled horizontally in Kubernetes can be a great way to monitor the growth of your business. However, monitoring large numbers of capsules becomes a challenge at times as users struggle with load balancers, noisy neighbors, resource contention, or other unexpected emergent properties of systems.

How to use outlier detection in Grafana Machine Learning?
With proper querying you will see that data visualized with outliers in yellow and a normality bar in blue. The sensitivity slider can then be used to adjust the thickness of this bar to adjust how extreme the data points need to be to be flagged as outliers.

Grafana alerting and outlier detection
For users to receive notifications when an outlier is detected in their data, Grafana Alerting can come in handy. Several sequential steps take users to the familiar home page to create an alert with a pre-configured relevant query.

Grafana Machine Learning currently supports the following data sources: Prometheus, Graphite, Loki (for metric queries only), Postgres, InfluxDB, BigQuery, Snowflake, Splunk, and Datadog.

The default usage limits are 1000 series per outlier detector and 10 outlier detectors per instance. Grafana stated that these limits can be increased upon request. Variance detection is available at no additional charge for Grafana Cloud customers with Pro, Advanced or Custom plans. Questions can be asked in the #machine-learning channel on the Grafana Labs Slack workspace.