Root cause: IncreaseAccumulator keeps only starting + last_seen and query(Increase) returns last − first (no reset detection), so any window spanning a counter restart is undercounted — often negative.
What causes a reset
A Prometheus/VM counter is in-process and goes back to 0 whenever the target restarts: deploys/rollouts, crashes/OOM-kills, autoscaling, node drains/preemptions, pod reschedules. (Counter overflow is also a "reset" but is rare with float64.) So a "reset within the window" simply means "the exporter restarted during [d]."
Testing
Drop this into the test module of asap-query-engine/src/precompute_operators/increase_accumulator.rs (it isolates the bug to the accumulator, which is exactly what MultipleIncrease stores per key):
#[test]
fn counter_reset_increase_is_wrong() {
use crate::data_model::Measurement;
use promql_utilities::query_logics::enums::Statistic;
// One counter, sampled 100 -> 150 -> (restart) 10 -> 60 within the window.
let mut acc = IncreaseAccumulator::new(Measurement::new(100.0), 0,
Measurement::new(100.0), 0);
acc.update(Measurement::new(150.0), 1_000);
acc.update(Measurement::new(10.0), 2_000); // reset: 150 -> 10
acc.update(Measurement::new(60.0), 3_000);
let inc = acc.query(Statistic::Increase, None).unwrap();
// BUG: last - first = 60 - 100 = -40 (negative!)
// VictoriaMetrics/Prometheus reset-correct this to +110.
assert_eq!(inc, -40.0);
}
#Run:
cargo test -p query_engine_rust --lib counter_reset_increase_is_wrong -- --nocapture
Expected Result
It passes asserting -40.0, which is the bug: rate shows the same (negative value_diff/time_diff). Prometheus/VictoriaMetrics' reset-corrected value here is +110 (its removeCounterResetsMaybeNaNs rebuilds the monotonic series [100,150,160,210] → 210−100).
Root cause:
IncreaseAccumulatorkeeps only starting + last_seen and query(Increase) returnslast − first(no reset detection), so any window spanning a counter restart is undercounted — often negative.What causes a reset
A Prometheus/VM counter is in-process and goes back to 0 whenever the target restarts: deploys/rollouts, crashes/OOM-kills, autoscaling, node drains/preemptions, pod reschedules. (Counter overflow is also a "reset" but is rare with float64.) So a "reset within the window" simply means "the exporter restarted during [d]."
Testing
Drop this into the test module of asap-query-engine/src/precompute_operators/increase_accumulator.rs (it isolates the bug to the accumulator, which is exactly what MultipleIncrease stores per key):
#[test]
#Run:
cargo test -p query_engine_rust --lib counter_reset_increase_is_wrong -- --nocapture
Expected Result
It passes asserting -40.0, which is the bug: rate shows the same (negative value_diff/time_diff). Prometheus/VictoriaMetrics' reset-corrected value here is +110 (its
removeCounterResetsMaybeNaNsrebuilds the monotonic series [100,150,160,210] → 210−100).