I enabled the recommended alerts on my AKS cluster. Every day I get an alert now that my DiskUsagePercentage exceeded the 80% threshold:
Metric name DiskUsedPercentage
Time Aggregation Average
Period Over the last 5 mins
Value 80.000186
Operator GreaterThan
Threshold 80
Criterion Type StaticThresholdCriterion
If I look at the AVG aggregation diskUsedPercentage for the node it's near constant at around 40%.
If I look at the MIN aggregation diskUsedPercentage for the node it's near constant at around 0.2%.
If I look at the MAX aggregation diskUsedPercentage for the node it's near constant at around 80%.
This doesn't make sense to me. When I read these numbers, it seems as though my disk used percentage is going up/down/up/down from 0.2% -> 80% each 5 min time bucket. That seems wrong to me. I have a small stable web application that doesn't write anything to disk. Not even logs.
What is going on here?