Repository navigation
IN LIST: retain short integer, floating-point, and fixed-size binary lists - #25187
geoffreyclaude wants to merge 2 commits into
Conversation
0186b14 to
ef1ad44
Compare
ef1ad44 to
11b0a61
Compare
11b0a61 to
41aaaa1
Compare
41aaaa1 to
46e59a1
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25187 +/- ##
==========================================
+ Coverage 82.60% 82.66% +0.06%
==========================================
Files 1145 1147 +2
Lines 444057 446510 +2453
Branches 444057 446510 +2453
==========================================
+ Hits 366796 369114 +2318
+ Misses 54996 54995 -1
- Partials 22265 22401 +136 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
46e59a1 to
d0ec00e
Compare
04a830c to
499119b
Compare
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 499119b (499119b) to bec8193 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "list=2/"
baseline:
ref: "bec81934341766b273dab7012ed127015d946cc6"
changed:
ref: "499119b2edbdecf40ce98fea090f50be6499750b"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 499119b (499119b) to bec8193 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "list=3/"
baseline:
ref: "bec81934341766b273dab7012ed127015d946cc6"
changed:
ref: "499119b2edbdecf40ce98fea090f50be6499750b"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 499119b (499119b) to bec8193 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "list=2/"
baseline:
ref: "bec81934341766b273dab7012ed127015d946cc6"
changed:
ref: "499119b2edbdecf40ce98fea090f50be6499750b"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 499119b (499119b) to bec8193 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "list=3/"
baseline:
ref: "bec81934341766b273dab7012ed127015d946cc6"
changed:
ref: "499119b2edbdecf40ce98fea090f50be6499750b"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
499119b to
4a5d1a4
Compare
4a5d1a4 to
6c57810
Compare
6c57810 to
202998f
Compare
202998f to
73d1e26
Compare
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(float32|float64|fixed_size_binary_(1|2|4|8|16))/.*/list=2/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(int32|int64|uint64|utf8|utf8view_inline)/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(float32|float64|fixed_size_binary_(1|2|4|8|16))/.*/list=3/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(int32|int64|uint64|utf8|utf8view_inline)/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(float32|float64|fixed_size_binary_(1|2|4|8|16))/.*/list=2/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 73d1e26 (73d1e26) to 3787a2b diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "^in_list_rewrite/(float32|float64|fixed_size_binary_(1|2|4|8|16))/.*/list=3/"
baseline:
ref: "3787a2b324288c743fb764bdb7b893d794b58b8c"
changed:
ref: "73d1e2609476d47af3c4d410d148722c4842736f"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
Which issue does this PR close?
Rationale for this change
DataFusion currently rewrites short
INlists likex IN (1, 2)asx = 1 OR x = 2. EvaluatingINdirectly can be faster.This PR keeps the
INform for the types below, so short lists can use DataFusion's existing optimizedINfilters.What changes are included in this PR?
INandNOT INlists forInt32,Int64,UInt64,Float32,Float64, andFixedSizeBinarycolumns with widths of 1, 2, 4, 8, or 16 bytes. The list must contain constant values of the column's type.INwith separate comparisons.What is the testing strategy for this PR?
Formatting, Clippy, the full lint suite, and extended workspace tests passed, including all 526 SQL logic files. Focused
in_list.sltand TPC-H checks also passed.The benchmark bot's ARM results show:
INis about 1.4–2.2× faster when no rows match or half the rows match. In the benchmark's extreme skew case—every row matches the first list value—INcan take up to about twice as long as separate comparisons. This PR accepts that tradeoff for the gains on batches with mixed matches or no matches. Results.INis faster in every measured case, including NULLs and small batches. Two-value results, three-value results.Benchmark setup
Both builds completed all 672 benchmark cases, covering 336 comparisons between the two forms. The baseline contains
mainat4d167a167plus the same benchmark code as the PR.Both forms are timed in each build. The timings cover evaluation of the filter expressions; planning happens before timing, and the optimizer is skipped to preserve both forms.
Are there any user-facing changes?
EXPLAINshowsINorNOT INfor these short lists in place of the expandedORorANDcomparisons.This performance PR aligns constant two- and three-value
Float32/Float64lists with existing behavior for 4+ values, including the signed-zero bug (+0.0versus-0.0). That bug predates this optimization series and already affects longer lists with constant or column values; fixing it is left to #25186.