Repository navigation
Support WindowGroupLimitExec (window-based limit pushdown) #4837
Description
Activity
This operator is a new TopK operator introduced in Spark 3.5
Before Spark 3.5
Catalyst rewrote this to something like Limit(3, Window(Sort(t))), which combined with TakeOrderedAndProjectExec gave you the classic heap-based top-K over a single stream. But this only worked when there was no PARTITION BY — the moment you added one, you fell back to the naive full-sort-full-window plan.Prob we can use
PartitionedTopKExecDF operator to handle thiswe currently addressing rank in Comet but we can migrate to apache/datafusion#22885
and dense_rank apache/datafusion#23869
Spark inserts
WindowGroupLimitExecto push a rank/row-number limit below a window (for exampleWHERE rn <= 10over a ranking window). Comet does not support this operator today, so plans containing it fall back to Spark.It is currently marked as planned in the operator support reference under Window.