Repository navigation
Commit 791660c
authored
fix: Estimate string equality and LIKE filter selectivity (#25835)
## Which issue does this PR close?
Closes #25614.
## Rationale for this change
String equality, LIKE, and NOT LIKE currently receive the same fallback
estimate. For a column with 1,000 rows, 200 NULLs, and 20 distinct
non-null values, equality should estimate 40 matching rows and
inequality 760, instead of estimating 200 for both.
## What changes are included in this PR?
- Estimate direct string equality/inequality using the distinct count
when available, under the existing uniform-distribution assumption.
- Classify LIKE patterns as literal, all-percent, or wildcard patterns.
Wildcard estimates use the configured fallback, halved for each anchored
end. Recognize escaped wildcards and Arrow's literal trailing-backslash
behavior.
- Complement negated estimates over non-null rows and propagate zero
null counts for LIKE operands. Literal ILIKE retains a heuristic because
case folding can merge distinct values.
- Retain existing estimates for compound or otherwise unsupported
expressions.
- Remove an unnecessary semicolon in the Windows rounding implementation
so the required Clippy check can run without that pre-existing warning.
## What is the testing strategy for this PR?
Public SQL planning tests in
`datafusion/core/tests/custom_sources_cases/statistics.rs` inject known
statistics through the existing table-provider fixture. They cover Utf8,
LargeUtf8, Utf8View, equality/inequality, pattern forms, escaped
characters, ILIKE, unknown distinct counts, and configured fallback
selectivity. Direct physical-plan tests cover patterns eliminated by SQL
simplification, NULL/empty inputs, and column-valued patterns.
Validation:
- 51 filter tests passed, including the unsupported-literal fallback.
- 3 new public SQL regression tests passed.
- `cargo fmt --all` passed.
- Targeted physical-plan Clippy passed with warnings denied.
- Ablation passed: all 3 new SQL tests fail with the original `main`
filter implementation, with the expected row-count assertion failures.
- TOML/Markdown formatting, license headers, typos, workflow checks, and
ASF YAML checks passed.
## Are there any user-facing changes?
String filter row estimates change. No public API changes. Wildcard
estimates remain heuristics in the absence of string histograms.1 parent 6a31aa8 commit 791660c
2 files changed
Lines changed: 443 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
285 | 285 | | |
286 | 286 | | |
287 | 287 | | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
288 | 410 | | |
289 | 411 | | |
290 | 412 | | |
| |||
0 commit comments