Repository navigation
Commit cee1dc7
authored
refactor: factor SortMergeJoinExec::execute into a reusable sort_merge_join_stream (#25490)
## Which issue does this PR close?
- No separate issue. Split out of #25217 so it can be reviewed on its
own.
## Rationale for this change
#25217 lets a hash join that runs out of memory finish as a sort-merge
join:
it sorts both inputs itself and then needs to run them through the same
join
streams `SortMergeJoinExec` uses. Today the only way to build those
streams is
inside `SortMergeJoinExec::execute`, which starts from two child
`ExecutionPlan`s rather than two already-sorted streams.
This PR separates "join two sorted streams" from "execute my children",
so the
first half can be reused. It is a refactor with no new caller yet; it is
split
out to keep the sort-merge join changes out of the (much larger) hash
join PR.
## What changes are included in this PR?
All in `sort_merge_join/exec.rs`:
- New `pub(crate) fn sort_merge_join_stream(SortMergeJoinInputs,
&metrics, &context)`.
It holds what `execute` did after executing its children: pick the
streamed
and buffered side by join type, register the (spillable) memory
consumer,
build the spill manager, and choose `BitwiseSortMergeJoinStream` or
`MaterializingSortMergeJoinStream` by join type family.
- New `pub(crate) struct SortMergeJoinInputs` carrying the two sorted
streams,
join keys, filter, join type, sort options, null equality, schema and
partition.
- `SortMergeJoinExec::execute` keeps the partition-count check, executes
both
children, calls the new function and applies the embedded projection.
Nothing is `pub`, so there is no public API change.
One ordering detail: `execute` used to call `execute()` on the streamed
child
first and the buffered child second; it now calls left then right,
because the
side selection moved into the shared function. For right-family joins
that
swaps the order of the two calls. Both still happen before anything is
polled.
## What is the testing strategy for this PR?
No behaviour change is intended, so this relies on the existing
`sort_merge_join` unit tests, which all pass unchanged. That includes
`stream_registers_as_a_spillable_consumer` and
`fair_spill_pool_leaves_room_for_the_streamed_sort` from #25250, which
check
that the memory consumer registered by the moved code is still
spillable.
## Are there any user-facing changes?
No.1 parent c607caf commit cee1dc7
1 file changed
Lines changed: 130 additions & 77 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
583 | 583 | | |
584 | 584 | | |
585 | 585 | | |
586 | | - | |
587 | | - | |
588 | | - | |
589 | | - | |
590 | | - | |
591 | | - | |
592 | | - | |
593 | | - | |
594 | | - | |
595 | | - | |
596 | | - | |
597 | | - | |
598 | | - | |
599 | | - | |
600 | | - | |
601 | | - | |
602 | | - | |
603 | | - | |
604 | 586 | | |
605 | | - | |
606 | | - | |
607 | | - | |
608 | | - | |
609 | | - | |
610 | | - | |
611 | | - | |
612 | | - | |
613 | | - | |
614 | | - | |
615 | | - | |
616 | | - | |
617 | | - | |
618 | | - | |
619 | | - | |
620 | | - | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
621 | 590 | | |
622 | | - | |
623 | | - | |
624 | | - | |
625 | | - | |
626 | | - | |
627 | | - | |
628 | | - | |
629 | | - | |
630 | | - | |
631 | | - | |
632 | | - | |
633 | | - | |
634 | | - | |
635 | | - | |
636 | | - | |
637 | | - | |
638 | | - | |
639 | | - | |
640 | | - | |
641 | | - | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
642 | 602 | | |
643 | | - | |
644 | | - | |
645 | | - | |
646 | | - | |
647 | | - | |
648 | | - | |
649 | | - | |
650 | | - | |
651 | | - | |
652 | | - | |
653 | | - | |
654 | | - | |
655 | | - | |
656 | | - | |
657 | | - | |
658 | | - | |
659 | | - | |
660 | | - | |
661 | | - | |
662 | | - | |
663 | | - | |
664 | | - | |
665 | | - | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
666 | 607 | | |
667 | 608 | | |
668 | 609 | | |
| |||
949 | 890 | | |
950 | 891 | | |
951 | 892 | | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
0 commit comments