Repository navigation
Conversation
…, off by default)
…threshold while its groups do not recur
…ting buckets of unique groups
github-actions
Bot
added
documentation
Contributor
Author
|
run benchmark clickbench_partitioned baseline: |
github-actions
Bot
added
common
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/partial-flush-controls (fe0866a) to 7f0ab42 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_COMPACTION: "false"
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"
DATAFUSION_EXECUTION_HASH_AGGREGATE_DETECT_REPEATED_GROUPS: "false"
DATAFUSION_EXECUTION_HASH_AGGREGATE_FINAL_BUCKETS: "false"
baseline:
ref: "7f0ab426eb84a967a93c42ee0bd6e928e49adeb6"Results will be posted here when complete File an issue against this benchmark runner |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #26098 +/- ##
==========================================
+ Coverage 82.64% 82.73% +0.09%
==========================================
Files 1147 1149 +2
Lines 445440 450009 +4569
Branches 445440 450009 +4569
==========================================
+ Hits 368123 372333 +4210
+ Misses 54970 54948 -22
- Partials 22347 22728 +381 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
|
run benchmark clickbench_partitioned baseline: |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/partial-flush-controls (fe0866a) to 7f0ab42 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_COMPACTION: "false"
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"
DATAFUSION_EXECUTION_HASH_AGGREGATE_DETECT_REPEATED_GROUPS: "false"
DATAFUSION_EXECUTION_HASH_AGGREGATE_FINAL_BUCKETS: "false"
baseline:
ref: "7f0ab426eb84a967a93c42ee0bd6e928e49adeb6"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/partial-flush-controls (fe0866a) to 7f0ab42 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_COMPACTION: "false"
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"
DATAFUSION_EXECUTION_HASH_AGGREGATE_DETECT_REPEATED_GROUPS: "false"
DATAFUSION_EXECUTION_HASH_AGGREGATE_FINAL_BUCKETS: "true"
baseline:
ref: "7f0ab426eb84a967a93c42ee0bd6e928e49adeb6"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/partial-flush-controls (fe0866a) to 7f0ab42 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_COMPACTION: "false"
DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD: "262144"
DATAFUSION_EXECUTION_HASH_AGGREGATE_DETECT_REPEATED_GROUPS: "false"
DATAFUSION_EXECUTION_HASH_AGGREGATE_FINAL_BUCKETS: "true"
baseline:
ref: "7f0ab426eb84a967a93c42ee0bd6e928e49adeb6"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
None. Benchmark-only experiment based on #26097 (
85b64a74cec1eab1e5a590234cd242fa0ece843d).Rationale for this change
Compare continuous Partial flushing with ordinary Final aggregation against main, and expose independent switches for follow-up experiments.
What changes are included in this PR?
88d36d4d) so string keys and string state are eligible again; preserve the original nested-state exclusions.hash_aggregate_bucket_compaction,hash_aggregate_detect_repeated_groups, andhash_aggregate_final_buckets.Requested comparison: all three switches false,
hash_aggregate_bucket_threshold=262144, against main at7f0ab426eb84a967a93c42ee0bd6e928e49adeb6. Existing skip-partial aggregation remains enabled. Main uses its ordinary behavior. This compares the full experiment with main, not just the incremental switch diff.What is the testing strategy for this PR?
The byte-only follow-up passed
cargo check -p datafusion-physical-plan, workspace format checks, full all-target/all-feature clippy with-D warnings, configuration documentation generation/format checks, andaggregate_bucketed.sltplusinformation_schema.slt. The aggregation test explicitly checks that exceeding a small group-count threshold does not flush a Partial table below 2 MiB.The following original validation and timing results apply to
fe0866a91, before removing the group-count trigger:Added SQLLogicTest coverage for integer, long StringView, and mixed keys with all three switches disabled, plus Final buckets enabled with compaction disabled. Updated configuration documentation and information_schema fixtures.
cargo fmt --alland format check: passed.cargo clippy --all-targets --all-features -- -D warnings: passed../ci/scripts/doc_prettier_check.sh --write --allow-dirty: passed.ECS comparison: Rust 1.99.0,
release-nonlto, 12 partitions on CPUs 0-11, batch size 8192, 99,997,497 partitioned input rows. Two reverse-order rounds; four iterations per cell, first excluded (six warm samples per variant). Unmodified benchmark runner, entire-query wall-time medians:Q33/Q34/Q35 gains repeat in both orders. Q10/Q11 regress slightly; Q31/Q32 remain near flat. The benchmark checks success and output row counts; SQLLogicTest provides value-level correctness coverage. These results compare the full experiment against the pinned current main; they do not isolate unrelated changes between the parent PR base and main.
Byte-only follow-up (
3ce7f0174): same ECS setup, all three switches false, two reverse-order rounds with six warm samples per variant. Compare the parent with the byte-only change and the pinned main in the same run.Are there any user-facing changes?
Three experimental configuration switches. The parent feature is disabled by default (
hash_aggregate_bucket_threshold=0). This PR is an experimental benchmark vehicle, not intended for merge.