Skip to content

rate/increase functions do no counter-reset correction #476

Description

@zaoxing

Root cause: IncreaseAccumulator keeps only starting + last_seen and query(Increase) returns last − first (no reset detection), so any window spanning a counter restart is undercounted — often negative.

What causes a reset
A Prometheus/VM counter is in-process and goes back to 0 whenever the target restarts: deploys/rollouts, crashes/OOM-kills, autoscaling, node drains/preemptions, pod reschedules. (Counter overflow is also a "reset" but is rare with float64.) So a "reset within the window" simply means "the exporter restarted during [d]."

Testing
Drop this into the test module of asap-query-engine/src/precompute_operators/increase_accumulator.rs (it isolates the bug to the accumulator, which is exactly what MultipleIncrease stores per key):

#[test]

fn counter_reset_increase_is_wrong() {
    use crate::data_model::Measurement;
    use promql_utilities::query_logics::enums::Statistic;

    // One counter, sampled 100 -> 150 -> (restart) 10 -> 60 within the window.
    let mut acc = IncreaseAccumulator::new(Measurement::new(100.0), 0,
                                           Measurement::new(100.0), 0);
    acc.update(Measurement::new(150.0), 1_000);
    acc.update(Measurement::new(10.0),  2_000); // reset: 150 -> 10
    acc.update(Measurement::new(60.0),  3_000);

    let inc = acc.query(Statistic::Increase, None).unwrap();
    // BUG: last - first = 60 - 100 = -40  (negative!)
    // VictoriaMetrics/Prometheus reset-correct this to +110.
    assert_eq!(inc, -40.0);
}

#Run:
cargo test -p query_engine_rust --lib counter_reset_increase_is_wrong -- --nocapture

Expected Result
It passes asserting -40.0, which is the bug: rate shows the same (negative value_diff/time_diff). Prometheus/VictoriaMetrics' reset-corrected value here is +110 (its removeCounterResetsMaybeNaNs rebuilds the monotonic series [100,150,160,210] → 210−100).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions