Add DeferredProjectionPager

README.md

@@ -629,6 +629,13 @@ App\Domain\Model\User:
 Dealing with pagination is a task that is often needed.
 This package contains some tools to help you with that:
 
+* `WindowCountPager` - the default: one query, the total counted with a window function.
+* `DeferredProjectionPager` - for rows that carry aggregated or joined objects.
+
+Both are honest implementations of `Pager` and both stay: pick the one that matches the width of your rows.
+
+## WindowCountPager
+
 ```php
 use Phpro\DbalTools\Pager\MappingPager;
 use Phpro\DbalTools\Pager\Pager;
@@ -652,6 +659,143 @@ $totalPages = $usersPager->totalPages();
 $users = [...$usersPager];
 ```
 
+## DeferredProjectionPager
+
+`WindowCountPager` adds `COUNT(1) OVER()` to the query it pages.
+PostgreSQL evaluates window functions before `ORDER BY`/`LIMIT` at the same query level, so every matching
+row is buffered before the page is cut.
+That is free for a narrow row, and expensive as soon as the row carries a `jsonb_build_object` or a
+`jsonb_agg_strict` projection over joined tables: those buffered rows are kilobytes wide and spill to temp
+files.
+
+`DeferredProjectionPager` pages *narrow* rows - the key column plus the count window - inside a CTE, and
+joins the fat projection onto that CTE.
+The total is still computed over the whole matching set, but out of `(key, total)` rows, and the expensive
+projection is only evaluated for the page that is returned.
+It remains a single statement, so there is one snapshot and no second round trip.
+
+**Which one to reach for:**
+
+* `WindowCountPager` for a narrow row. It is correct, cheaper, and pays nothing for a CTE it does not need.
+* `DeferredProjectionPager` when a row carries aggregated or joined objects.
+
+```php
+use Doctrine\DBAL\Query\QueryBuilder;
+use Phpro\DbalTools\Expression\Alias;
+use Phpro\DbalTools\Expression\Factory\NamedParameter;
+use Phpro\DbalTools\Expression\ILike;
+use Phpro\DbalTools\Expression\JsonbAggStrict;
+use Phpro\DbalTools\Expression\JsonbBuildObject;
+use Phpro\DbalTools\Expression\OrderBy;
+use Phpro\DbalTools\Pager\DeferredProjectionPager;
+use Phpro\DbalTools\Pager\MappingPager;
+use Phpro\DbalTools\Pager\Pagination;
+use Phpro\DbalTools\Query\CompositeQuery;
+
+// Which rows: the key column and every filter. No display columns, no join that only adds data.
+$keys = $connection->createQueryBuilder();
+$matchingKeys = new CompositeQuery(
+    $connection,
+    $keys
+        ->select(UsersTableColumns::Id->select())
+        ->from(UsersTable::name())
+        ->where(new ILike(
+            UsersTableColumns::Username->column(),
+            NamedParameter::createForTableColumn($keys, UsersTableColumns::Username, 'jo%', ':username'),
+        )->toSQL()),
+    [],
+);
+
+$usersPager = new MappingPager(
+    DeferredProjectionPager::create(
+        pagination: new Pagination(page: $page, limit: $limit),
+        matchingKeys: $matchingKeys,
+        key: UsersTableColumns::Id->column(),
+        order: new OrderBy(OrderBy::field(UsersTableColumns::Username->column(), OrderBy::ASC)),
+        // What a row looks like: a new query from the table of the key.
+        // This one needs no CTE, so it ignores both arguments.
+        projection: static fn (CompositeQuery $folded, string $pageAlias): QueryBuilder => $connection->createQueryBuilder()
+            ->select(
+                ...UsersTable::columns()->select(),
+                ...[new Alias(
+                    JsonbAggStrict::onManyLeftJoinedJsonObjects(
+                        new JsonbBuildObject([
+                            'id' => PostsTableColumns::Id->column(),
+                            'post' => PostsTableColumns::Post->column(),
+                        ]),
+                        PostsTableColumns::Id->column(),
+                    ),
+                    'posts',
+                )->toSQL()],
+            )
+            ->from(UsersTable::name())
+            ->leftJoin(...UsersTable::joinOntoPosts())
+            ->groupBy(UsersTableColumns::Id->use()),
+    ),
+    $userMapper,
+);
+```
+
+For page 2 with a limit of 10, that runs as one statement. The main query of `$matchingKeys` became the
+`deferred_page` CTE, with the count window, the order and the page added to it. The projection became the
+new main query, joined onto that CTE:
+
+```sql
+WITH deferred_page AS (
+    SELECT users.user_id, COUNT(1) OVER() AS total_results
+    FROM users
+    WHERE users.username ILIKE :username
+    ORDER BY users.username ASC
+    LIMIT 10 OFFSET 10
+)
+SELECT
+    users.user_id, users.username, users.first_name, users.last_name,
+    jsonb_agg_strict(
+        CASE WHEN posts.post_id IS NULL THEN NULL
+        ELSE jsonb_build_object('id', posts.post_id, 'post', posts.post) END
+    ) AS posts,
+    (SELECT deferred_page.total_results FROM deferred_page LIMIT 1) AS total_results
+FROM users
+LEFT JOIN posts posts ON users.user_id = posts.user_id
+INNER JOIN deferred_page deferred_page ON users.user_id = deferred_page.user_id
+GROUP BY users.user_id
+ORDER BY users.username ASC
+```
+
+You describe the list twice:
+
+* `matchingKeys` decides which rows are in the list: a composite whose main query selects the key column
+  and nothing else, with every filter and scope join. No display columns, order or limit, the pager adds
+  those.
+* `key` is that column, table qualified. The page is joined back to the projection on it.
+* `projection` decides what a row looks like: a new query builder selecting everything a row shows from the
+  table of the key, with its joins, aggregates and group by, and no filters, order or limit. The closure
+  receives the composite that will run and the name of the page CTE. Use the composite to join a CTE
+  registered on `matchingKeys` or to register one of your own; a projection that needs no CTE ignores both
+  arguments.
+
+Things worth knowing:
+
+* The filters and the scope joins belong on `$matchingKeys`. A filter applied by the projection would drop
+  rows after the total was computed, so the page would come back short and the total would over-report.
+* To filter on a related table, for example users with a matching post, use `EXISTS` on `$matchingKeys`
+  rather than a join: a join yields the key once per matching post. To limit which related rows the
+  projection aggregates, put the condition in its `LEFT JOIN`, not in a `WHERE`, so rows without a match
+  stay on the page.
+* Every CTE you registered on `$matchingKeys` survives the fold, and so does a parameter bound on its main
+  query: that same query builder is moved into the `WITH` list, and `CompositeQuery::execute()` merges the
+  parameters of every registered builder.
+* The key must be table qualified (the join is derived from it) and unique among the matching keys. A
+  duplicate multiplies the projected rows and makes the total disagree with the page.
+* Aggregate freely. The pager reads its count as a scalar sub-query rather than as a column of the joined
+  CTE, so a projection that groups needs no group-by for it.
+* The pager owns the join onto the page CTE, the count column, and the order - which it applies to both
+  levels, since the narrow sort decides *which* rows the page holds and the outer one the order they come
+  back in. The closure only supplies its projection.
+* The projection closure **returns** its query rather than writing onto the folded main query, because
+  `QueryBuilder` keeps its select, from and join parts private with no setters.
+* Rows are yielded verbatim, the count field included, exactly like `WindowCountPager`.
+
 # Building queries
 
 This package contains a set of tools that helps you build queries based on partial SQL Expressions.

src/DbalToolsBundle.php

@@ -8,8 +8,16 @@
 use Symfony\Component\DependencyInjection\Loader\Configurator\ContainerConfigurator;
 use Symfony\Component\HttpKernel\Bundle\AbstractBundle;
 
+/**
+ * @psalm-suppress DeprecatedInterface - Symfony 8.1 deprecates HttpKernel's BundleInterface, its replacement does not exist before 8.1.
+ * @psalm-suppress UnusedPsalmSuppress - Only needed for Symfony 8.1 and up.
+ */
 final class DbalToolsBundle extends AbstractBundle
 {
+    /**
+     * @psalm-suppress ParamNameMismatch - Symfony 8.1 renamed these parameters, the names cannot match both 7.4/8.0 and 8.1.
+     * @psalm-suppress UnusedPsalmSuppress - Only needed for Symfony 8.1 and up.
+     */
     public function loadExtension(array $config, ContainerConfigurator $container, ContainerBuilder $builder): void
     {
         $configDir = __DIR__.'/../config';

src/Pager/DeferredProjectionPager.php

@@ -0,0 +1,199 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Phpro\DbalTools\Pager;
+
+use Doctrine\DBAL\Query\QueryBuilder;
+use Phpro\DbalTools\Column\Column;
+use Phpro\DbalTools\Expression\Alias;
+use Phpro\DbalTools\Expression\Count;
+use Phpro\DbalTools\Expression\OrderBy;
+use Phpro\DbalTools\Expression\Over;
+use Phpro\DbalTools\Expression\SqlExpression;
+use Phpro\DbalTools\Query\CompositeQuery;
+use Psl\Iter\Iterator;
+use Psl\Str;
+use function Psl\invariant;
+use function Psl\Iter\first;
+use function Psl\Math\ceil;
+use function Psl\Vec\map;
+
+/**
+ * Pages narrow rows and projects the fat ones only for the page it returns, in one statement.
+ *
+ * Reach for it when a row carries aggregated or joined objects; {@see WindowCountPager} stays the right
+ * pager for a narrow row, which pays nothing for a CTE it does not need.
+ *
+ * PostgreSQL evaluates window functions before `ORDER BY`/`LIMIT` at the same query level, so
+ * `COUNT(1) OVER()` buffers every matching row before the page is cut. Over a `jsonb_build_object`
+ * projection those buffered rows are kilobytes wide and spill to temp files. Here the count window sits on
+ * a narrow query selecting `$key` alone, which is then folded into a CTE and joined back to the fat
+ * projection: the true total is still computed over the whole matching set, but out of `(key, total)` rows.
+ *
+ * @template-implements Pager<array<string, mixed>>
+ */
+final readonly class DeferredProjectionPager implements Pager
+{
+    /**
+     * Fixed rather than a parameter: a caller registering a CTE of this name fails loudly at parse time,
+     * which is the good failure, where a parameter would be one more thing every call site passes.
+     */
+    private const string PAGE_ALIAS = 'deferred_page';
+
+    /**
+     * @param non-empty-string                    $countField
+     * @param Iterator<int, array<string, mixed>> $iterator
+     */
+    private function __construct(
+        private Pagination $pagination,
+        private Iterator $iterator,
+        private string $countField,
+    ) {
+    }
+
+    /**
+     * You describe the list twice: which rows are in it, and what a row looks like.
+     *
+     * `$matchingKeys` decides which rows are in the list. Its main query selects `$key` and nothing else,
+     * and carries every filter and scope join, for example `SELECT users.id FROM users WHERE ...`. Leave
+     * out display columns, order and limit: the pager adds the total, the order and the page. Every CTE
+     * registered on it stays available to the projection.
+     *
+     * `$key` is the column that main query selects, table qualified (`users.id`). The page is joined back
+     * to the projection on it, so it must be unique per row: a duplicate multiplies the projected rows and
+     * the total disagrees with the page.
+     *
+     * `$order` sorts the list. It is applied to both levels, since the narrow sort decides which rows the
+     * page holds and the outer one the order they come back in.
+     *
+     * `$projection` decides what a row looks like. Return a new query builder that selects everything a row
+     * shows `FROM` the table of `$key`, including any joins, aggregates and group by it needs. Leave out
+     * filters, order and limit: the pager joins it onto the page, and a filter here would drop rows after
+     * the total was counted. The closure receives the composite that will run and the name of the page
+     * CTE. A projection that needs a CTE joins one registered on `$matchingKeys` through the composite, or
+     * registers its own on it; one that needs none ignores both arguments.
+     *
+     * @param \Closure(CompositeQuery, non-empty-string): QueryBuilder $projection
+     * @param non-empty-string                                         $countField
+     */
+    public static function create(
+        Pagination $pagination,
+        CompositeQuery $matchingKeys,
+        Column $key,
+        OrderBy $order,
+        \Closure $projection,
+        string $countField = 'total_results',
+    ): self {
+        invariant(null !== $key->from, 'The page key must be table qualified, so the projection join can be derived.');
+
+        // First clone the composite to avoid updates on the provided query.
+        $matchingKeys = clone $matchingKeys;
+        $narrow = $matchingKeys->mainQuery();
+        $narrow->addSelect(
+            new Alias(
+                Over::aggregation(new Count(SqlExpression::int(1)), Over::fullWindow()),
+                $countField,
+            )->toSQL(),
+        );
+        self::applyOrder($narrow, $order);
+        $narrow->setMaxResults($pagination->limit);
+        $narrow->setFirstResult(($pagination->page - 1) * $pagination->limit);
+
+        /**
+         * The narrow query becomes a CTE and the main query a fresh empty one, with every CTE the caller
+         * registered preserved. A parameter bound on the narrow query survives this: the very same builder
+         * moves into the `WITH` list, and `execute()` merges every registered CTE builder's parameters.
+         */
+        $folded = $matchingKeys->moveMainQueryToSubQuery(self::PAGE_ALIAS);
+
+        $fat = $projection($folded, self::PAGE_ALIAS);
+
+        /**
+         * The count is read as a scalar sub-query rather than as a column of the joined CTE, so that a
+         * projection which aggregates does not have to group by it. A plain column reference would be
+         * neither aggregated nor functionally dependent on the group key, which PostgreSQL rejects, and
+         * wrapping it in `MAX()` is not an option either: that would turn a non-aggregating projection
+         * into an aggregate one and collapse the page to a single row. The value is identical on every row
+         * of the page, so reading one is exact. The sub-query is the second reference to a CTE holding one
+         * page of narrow rows, which costs nothing.
+         */
+        $fat->addSelect(
+            new Alias(
+                SqlExpression::parenthesized(sprintf(
+                    'SELECT %s FROM %s LIMIT 1',
+                    $folded->cteColumn(self::PAGE_ALIAS, $countField)->toSQL(),
+                    self::PAGE_ALIAS,
+                )),
+                $countField,
+            )->toSQL(),
+        )->innerJoin(...$folded->joinOntoCte(
+            self::PAGE_ALIAS,
+            $key->from,
+            $key,
+        ));
+        self::applyOrder($fat, $order);
+
+        /**
+         * The closure returns its projection rather than writing onto the empty main query, because
+         * `QueryBuilder` keeps its select, from and join parts private with no setters: a pre-built query
+         * cannot be transplanted, so it is installed wholesale instead.
+         */
+        $hydrated = $folded->map(static fn (QueryBuilder $mainQuery): QueryBuilder => $fat);
+
+        return new self(
+            $pagination,
+            Iterator::from(
+                /**
+                 * @return iterable<int, array<string, mixed>>
+                 */
+                static function () use ($hydrated): iterable {
+                    yield from $hydrated->execute()->iterateAssociative();
+                },
+            ),
+            $countField,
+        );
+    }
+
+    public function pagination(): Pagination
+    {
+        return $this->pagination;
+    }
+
+    public function totalResults(): int
+    {
+        $first = first($this->iterator) ?? [];
+
+        return (int) ($first[$this->countField] ?? 0);
+    }
+
+    public function totalPages(): int
+    {
+        $totalResults = $this->totalResults();
+        $limit = $this->pagination()->limit;
+        if (!$totalResults) {
+            return 1;
+        }
+
+        return (int) ceil($totalResults / $limit);
+    }
+
+    public function traverse(\Closure $mapper): iterable
+    {
+        return map($this, $mapper);
+    }
+
+    public function getIterator(): \Traversable
+    {
+        yield from $this->iterator;
+    }
+
+    /**
+     * `OrderBy` renders its own `ORDER BY` keyword, which `QueryBuilder::orderBy()` prepends in turn; the
+     * builder joins its parts with a comma, so the whole expression goes in as one part.
+     */
+    private static function applyOrder(QueryBuilder $query, OrderBy $order): void
+    {
+        $query->orderBy(Str\strip_prefix($order->toSQL(), 'ORDER BY '));
+    }
+}