Skip to content

Commit a2b8093

Browse files
authored
feat: simplify contradicting and redundant predicates on a column (#25207)
## Which issue does this PR close? - Closes #5830. ## Rationale for this change `simplify_predicates` reduces the `>`/`>=` and the `<`/`<=` comparisons on a column to their most restrictive bound, but never compares the two groups with each other, and only looks for contradictions between equalities. A filter no row can satisfy therefore still scans the table. | `WHERE` clause | Before | After | | --- | --- | --- | | `a > 3 AND a < 1` | `a > 3 AND a < 1` | `EmptyRelation` | | `a > 1 AND a < 1` | `a > 1 AND a < 1` | `EmptyRelation` | | `a >= 1 AND a < 1` | `a >= 1 AND a < 1` | `EmptyRelation` | | `a = 7 AND a < 2` | `a = 7 AND a < 2` | `EmptyRelation` | | `a = 7 AND a != 7` | `a = 7 AND a != 7` | `EmptyRelation` | | `a = 7 AND a > 5` | `a = 7 AND a > 5` | `a = 7` | | `a > 10 AND a != 5` | `a > 10 AND a != 5` | `a > 10` | `a >= 1 AND a <= 1` still simplifies to `a = 1`: bounds meeting at a value both admit stay satisfiable. ## What changes are included in this PR? - `fix`: skip comparisons against a NULL literal when grouping. `ScalarValue::try_cmp` follows sort order, where NULL sits below every other value, so `a > NULL AND a > 5` was reduced to `a > 5` although `a > NULL` is never true. Reachable only through the public `simplify_predicates`, as `SimplifyExpressions` folds these first. - `feat`: compare the groups with each other. Bounds leaving no value, and an equality that contradicts another predicate, reduce the conjunction to `false`; an equality drops what it subsumes; `!=` joins the analysis and is dropped once a bound excludes its value. - `test`: unit tests and `sqllogictest` cases. Reducing to `false` is valid here because a `Filter` keeps a row only when its predicate is true, making NULL and `false` interchangeable. ## What is the testing strategy for this PR? Unit tests in `simplify_predicates.rs` cover each comparison operator against an equality, every combination of strict and inclusive bounds, `!=` dropped and kept, and comparisons against NULL. `simplify_predicates.slt` checks the plans and drops two `# TODO` markers this PR implements. ## Are there any user-facing changes? Filters no row can satisfy no longer scan their input. Result sets and public APIs are unchanged.
1 parent 5636128 commit a2b8093

5 files changed

Lines changed: 444 additions & 53 deletions

File tree

‎datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2909,6 +2909,13 @@ mod tests {
29092909
let result = simplify(expr.clone());
29102910
// The expression should not have been simplified
29112911
assert_eq!(result, expr);
2912+
2913+
// Signed zeros compare equal at runtime and must not be treated as
2914+
// different literals
2915+
let expr = col("f")
2916+
.eq(lit(0.0_f64))
2917+
.and(col("f").not_eq(lit(-0.0_f64)));
2918+
assert_eq!(simplify(expr.clone()), expr);
29122919
}
29132920

29142921
#[test]

0 commit comments

Comments
 (0)