test(lower): real-world PromQL conformance over the awesome-prometheus-alerts corpus - #32
Merged
Merged
Conversation
…s-alerts corpus Adds a 949-query corpus of real-world alerting PromQL expressions (deduplicated `expr` values from https://samber.github.io/awesome-prometheus-alerts/, MIT) plus a conformance suite over it. Corpus-wide invariants: - Totality — every real query lowers to Ok or a clean LoweringError, never panics. - Parseability — 0 of the 949 fail at the parse stage (the parser accepts 100% of real-world alert PromQL). - A lowering coverage floor (ratchet) guards against regressions. Targeted tests pin the L3 shape of the patterns we lower (vector-vs-vector comparisons; and the bodies of the histogram-quantile latency SLO, error-rate ratio, and `sum by (job) (up)` patterns), and `__GAP` tests pin the dominant patterns we cleanly reject today so support flips them deliberately: - scalar-threshold comparisons (`… > 0` / `… != 1`) — ~822/949 of the corpus; rejected only because a bare numeric operand has no L2 scalar node yet. This is the single biggest blocker to lowering real alerts. - `absent` / `changes` / `delta` / `predict_linear` / `vector` functions. - `without(...)` grouping. No production code changes — tests only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
test/awesome-prometheus-alerts-corpus
branch
from
July 2, 2026 13:00
d5ca25e to
aeee242
Compare
This was referenced Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds PromQL parsing/lowering conformance tests over real-world alerting queries — the 949 deduplicated
exprvalues from awesome-prometheus-alerts (MIT), spanning Prometheus self-monitoring, host/hardware, node-exporter, databases, brokers, Kubernetes, and more.Tests only — no production code changes. The corpus snapshot lives in
crates/lower/tests/data/awesome_prometheus_alerts.txt; the suite iscrates/lower/tests/awesome_prometheus_alerts.rs.What it pins
Corpus-wide invariants (
corpus_lowering_is_total_and_fully_parseable):Okor a cleanLoweringError, never panics.Pattern shape tests (verbatim corpus queries) — vector-vs-vector comparisons →
BinaryOp{Compare}, and the bodies of the canonical patterns lower to the right L3: histogram-quantile latency SLO (Quantile/sum by (le)/rate), error-rate ratio (BinaryOp(Div)of twosum(rate)),sum by (job) (up).__GAPtests pin the dominant patterns we cleanly reject today (so adding support flips them deliberately):… > 0,… != 1) — ~822/949 of the corpus, rejected only because a bare numeric operand has no L2 scalar node yet. This is the single biggest blocker to lowering real alerts.absent/changes/delta/predict_linear/vectorfunctions.without(...)grouping.Headline finding
Real alerting PromQL is overwhelmingly
<vector-expr> <cmp> <scalar>. Parsing is already 100%; lowering coverage is gated almost entirely by the scalar-threshold operand gap — the bodies underneath (rate,*_over_time,histogram_quantile,sum by (…)) lower fine on their own. That suggests a high-leverage follow-up: a scalar/number-literal node in the L2 PromQL path would unlock a large fraction of the corpus.Notes
cargo clippy --all-targetsclean.main; does not touch or conflict with feat(lower): arbitrary PromQL function nesting in aggregates (#27) #28/feat(lower): nested SQL query functions via derived tables (#27) #29/fix(lower): topk over a bare selector preserves by-labels (#30) #31. (The__GAPcategories chosen here are not affected by those PRs, so the assertions stay stable regardless of merge order.)🤖 Generated with Claude Code