fix(joins): return joined internal attributes for alias.* and joined orders

select(['alias.*']) stood for the join's implicit projection (the joined
$id and attributes), so it never returned the joined $sequence, $createdAt,
$updatedAt or $permissions a direct read of the joined collection returns.
On a left, right or full outer join it also lost alias.$id: the Database
layer treated alias.* as not selecting the joined $id, selected it only to
tell unmatched rows apart and stripped it from the result, so the next page
threw "Cursor has no value for order attribute 'alias.$id'".

A read ordered by a joined internal attribute (alias.$sequence,
alias.$createdAt) without a named select could not page either: the
implicit projection leaves joined internals out, so no returned row carried
the order value its cursor needs. select(['*', 'alias.$createdAt']) did not
help because any '*' replaced the whole select with the implicit projection,
silently dropping the named joined columns.

alias.* now returns the joined row as a direct read does: $id, $sequence,
$createdAt, $updatedAt and $permissions next to the attributes. $tenant stays
out (it is the read's own tenant, and callers strip only the main $tenant
before responding); it remains selectable by name. A read without a select
or with '*' projects the joined columns its order names and the joined
columns named next to '*', on every SQL engine and in both halves of the
UNION ALL full outer join emulation, so its rows page in both directions.
Only the projection changes: the order, tie keys and the MariaDB bounded
join page are untouched, so no read gains a sort. The implicit projection
without such an order keeps its documented shape. An unmatched outer row's
alias.$permissions is null like its other joined values instead of [].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Author: abnegate

CHANGELOG.md

@@ -70,19 +70,22 @@ have to make, with the 7.x and 8.0 forms side by side.
   - A join without a select returns the main document plus, under each join alias, the joined collection's `$id`
     and attributes as `alias.$id` and `alias.attribute`. The joined collection's internal attributes (`$tenant`,
     `$permissions`, `$sequence`, `$createdAt`, `$updatedAt`) and its relationship attributes that hold a column (the
-    side that stores the related document's id) are returned only when a select names them. Joined values are never
-    returned under a bare attribute name.
-  - `select('alias.*')` next to other selects returns the joined `$id` and attributes, as a join without a select
-    does. An order may name a joined attribute by its bare name when only one join's collection declares it and the
-    main collection does not; a name several joins declare throws `Exception\Query`.
+    side that stores the related document's id) are returned only when a select names them, or, for the internal
+    attributes other than `$tenant`, when the select names `alias.*` or the read orders by them. Joined values are
+    never returned under a bare attribute name.
+  - `select('alias.*')`, alone or next to other selects, returns the joined row as a direct read of the joined
+    collection returns it: its `$id`, `$sequence`, `$createdAt`, `$updatedAt`, `$permissions` and attributes, but not
+    its `$tenant`. Joined columns named next to `*` (`select(['*', 'alias.$createdAt'])`) are returned with
+    everything `*` returns. An order may name a joined attribute by its bare name when only one join's collection
+    declares it and the main collection does not; a name several joins declare throws `Exception\Query`.
   - Joined attributes are returned as a direct read of the joined collection returns them: cast to their types and
     passed through every decode filter they declare, so encrypted attributes are decrypted, JSON and arrays decoded
     and datetimes formatted. This applies to the implicit projection and to `select('alias.attribute')` alike. A
     decode filter receives a document built from the joined row: `$id`, `$collection` and the joined attributes the
-    query returned (`$sequence` and the other internal attributes only when selected). When an outer join matches no
-    row, its attributes are null, whether or not the select names `alias.$id` (outside `distinct()` reads, which
-    select only what they name). A cursor taken from a joined result can be passed back with `cursorAfter()` or
-    `cursorBefore()`: its joined values are encoded with the joined collection's filters.
+    query returned (`$sequence` and the other internal attributes only when the query returns them). When an outer
+    join matches no row, its attributes are null, whether or not the select names `alias.$id` (outside `distinct()`
+    reads, which select only what they name). A cursor taken from a joined result can be passed back with
+    `cursorAfter()` or `cursorBefore()`: its joined values are encoded with the joined collection's filters.
   - A column under a join alias (`alias.column`) must be valid on the joined collection for the query type it is
     used in: an attribute the joined collection declares, or an internal attribute the query type accepts on the
     main collection (`alias.$permissions` can be selected but not filtered or ordered by, and `alias.$collection` is

UPGRADE.md

@@ -1345,9 +1345,16 @@ API might expect. [CHANGELOG.md](CHANGELOG.md) describes the features themselves
 - A select of joined attributes returns what it names. The main collection's unselected attributes are left out, as
   for any select; only a select of a related document's attributes (`relationship.attribute`) returns the others as
   well, as in 7.x.
-- A select may name `alias.*` next to other selects: it stands for what the join returns without a select, the
-  joined collection's `$id` and attributes as `alias.$id` and `alias.attribute`. An aggregation query still rejects
-  it as an ungrouped select, and an alias the query does not join is not found.
+- A join without a select, or with `select(['*'])`, returns under each join alias the joined collection's `$id` and
+  attributes as `alias.$id` and `alias.attribute`, plus each joined internal attribute the read orders by
+  (`alias.$sequence`, `alias.$createdAt`, `alias.$updatedAt`), so its rows can be passed back as a cursor.
+- A select may name `alias.*`, alone or next to other selects: it returns the joined row as a direct read of the
+  joined collection returns it, its `$id`, `$sequence`, `$createdAt`, `$updatedAt`, `$permissions` and attributes as
+  `alias.$id`, `alias.$sequence`, ... and `alias.attribute`. It never returns the joined `$tenant`: select
+  `alias.$tenant` to read it. A row an outer join left unmatched holds null for each of them. An aggregation query
+  still rejects `alias.*` as an ungrouped select, and an alias the query does not join is not found.
+- Joined columns named next to `*` are returned with everything `*` returns: `select(['*', 'alias.$createdAt'])`
+  returns the main document, the joined `$id` and attributes, and `alias.$createdAt`.
 - An order may name a joined attribute by its bare name when the main collection does not declare it and exactly one
   join's collection does (`orderAsc('price')` over a join whose collection declares `price`); a name the main
   collection declares always orders by the main collection. A bare name more than one join declares throws
@@ -1397,7 +1404,9 @@ API might expect. [CHANGELOG.md](CHANGELOG.md) describes the features themselves
 - Pass a row the same read returned as the cursor. It has to carry every value the read orders by, under the name the
   read orders by (`note.score`, `$sequence`, `note.$id`); a missing value throws `Utopia\Database\Exception\Order`
   (`Cursor has no value for order attribute 'note.$id'. …`). A value is never taken from the main document's attribute
-  of the same name. A read whose `select()` leaves a paged join's `alias.$id` out has to select it to be paged.
+  of the same name. A read whose `select()` names attributes without `*` has to select every joined value it orders
+  by, a paged join's `alias.$id` included, to be paged; `alias.*` selects them all. A read without a select, or
+  with `*`, returns them.
   `cursor()` and `iterate()` check the last row of each full batch before yielding the batch, so such a read throws
   before the first row; a read that fits in one batch is not paged and needs no such value.
 - A value may be null (a row an outer join did not match, a nullable attribute). Nulls keep the engine's position:

src/Database/Adapter/SQL.php

@@ -88,6 +88,17 @@ abstract class SQL extends Adapter implements Feature\RawQuery, Feature\QueryBui
 
     private const string FOJ_ROWS_ALIAS = 'foj_rows';
 
+    /**
+     * The internal attributes `alias.*` returns next to the joined `$id`: those a direct read of the joined
+     * collection returns, but `$tenant`, which is the read's own tenant on every joined row.
+     */
+    private const array JOINED_ROW_INTERNALS = [
+        Document::SEQUENCE,
+        Document::CREATED_AT,
+        Document::UPDATED_AT,
+        Document::PERMISSIONS,
+    ];
+
     /**
      * MariaDB, MySQL and SQLite accept OFFSET only after a LIMIT; this one bounds nothing on any engine.
      */
@@ -1706,6 +1717,7 @@ public function find(Document $collection, array $queries = [], ?int $limit = 25
                 $alias,
                 $roles,
                 $forPermission,
+                orderAttributes: $orderAttributes,
             );
             $this->applyFullOuterJoinOrderProjection(
                 $left,
@@ -1739,6 +1751,7 @@ public function find(Document $collection, array $queries = [], ?int $limit = 25
                 $alias,
                 $roles,
                 $forPermission,
+                orderAttributes: $orderAttributes,
             );
             $this->applyFullOuterJoinOrderProjection(
                 $right,
@@ -1784,6 +1797,7 @@ public function find(Document $collection, array $queries = [], ?int $limit = 25
                 $alias,
                 $roles,
                 $forPermission,
+                orderAttributes: $orderAttributes,
             );
 
             $vectorDistance = null;
@@ -4075,11 +4089,11 @@ protected function executeUpsertBatch(
      * database column names (like _uid, _id) and ensures internal columns
      * are always included.
      *
-     * An `alias.*` selection stands for the columns the join returns without a select, from $joinSelections.
+     * An `alias.*` selection stands for the joined columns $joinSelections lists under that alias.
      *
      * @param  array<string>  $selections
      * @param  array<string>  $joinAliases
-     * @param  array<string, list<string>>  $joinSelections  The selections a read without a select makes under each join alias
+     * @param  array<string, list<string>>  $joinSelections  The selections `alias.*` makes under each join alias
      */
     private function applySelectionProjection(
         SQLBuilder $builder,
@@ -4273,21 +4287,34 @@ protected function mapSelectionsToColumns(array $selections, bool $includeIntern
     }
 
     /**
-     * The projection of a join without a select: every column of the main table, and under each join
-     * alias the joined collection's `$id` and the attributes the Database layer handed over for it.
-     * A joined table's internal columns are returned only when a select names them.
+     * The projection of a join without a select or with `*`: every column of the main table, and under each
+     * join alias the joined collection's `$id` and the attributes the Database layer handed over for it. A
+     * joined table's internal columns are returned only when a select names them or when the read orders by
+     * them, so that every row it returns can be passed back as its cursor.
      *
      * @param  list<array{table: string, alias: string}>  $joinTablePrefixes
+     * @param  array<string>  $additions  Selections next to `*` and order attributes; those under a join alias are projected too
      */
-    private function applyJoinProjection(SQLBuilder $builder, Document $collection, array $joinTablePrefixes, string $alias): void
+    private function applyJoinProjection(SQLBuilder $builder, Document $collection, array $joinTablePrefixes, string $alias, array $additions = []): void
     {
         $builder->select([$this->filter($alias).'.*']);
 
+        $joinAliases = \array_column($joinTablePrefixes, 'alias');
+        $aliasSet = \array_fill_keys($joinAliases, true);
+        $selections = \array_merge(...\array_values($this->joinSelections($collection, $joinTablePrefixes)));
+        foreach ($additions as $addition) {
+            $dot = \strpos($addition, '.');
+            if ($dot !== false && isset($aliasSet[\substr($addition, 0, $dot)])) {
+                $selections[] = $addition;
+            }
+        }
+
         $this->applySelectionProjection(
             $builder,
-            \array_merge(...\array_values($this->joinSelections($collection, $joinTablePrefixes))),
+            $selections,
             includeInternal: false,
-            joinAliases: \array_column($joinTablePrefixes, 'alias'),
+            joinAliases: $joinAliases,
+            joinSelections: $this->joinWildcardSelections($collection, $joinTablePrefixes),
         );
     }
 
@@ -4317,6 +4344,26 @@ private function joinSelections(Document $collection, array $joinTablePrefixes):
         return $selections;
     }
 
+    /**
+     * What `alias.*` selects under each join alias: what a read without a select returns there, and the joined
+     * collection's internal attributes a direct read of it returns.
+     *
+     * @param  list<array{table: string, alias: string}>  $joinTablePrefixes
+     * @return array<string, list<string>>
+     */
+    private function joinWildcardSelections(Document $collection, array $joinTablePrefixes): array
+    {
+        $selections = $this->joinSelections($collection, $joinTablePrefixes);
+        foreach ($selections as $joinAlias => $columns) {
+            foreach (self::JOINED_ROW_INTERNALS as $internal) {
+                $columns[] = $joinAlias.'.'.$internal;
+            }
+            $selections[$joinAlias] = $columns;
+        }
+
+        return $selections;
+    }
+
     /**
      * Map Database type constants to Schema Table column definitions.
      *
@@ -4807,6 +4854,7 @@ private function rewriteFullOuterJoins(array $queries, Method $replacement): arr
      * @param  list<array{table: string, alias: string}>  $joinTablePrefixes
      * @param  array<Query>  $adapterFilterQueries
      * @param  array<string>  $roles
+     * @param  array<string>  $orderAttributes  The attributes the read orders by
      */
     private function configureFindBuilder(
         SQLBuilder $builder,
@@ -4821,6 +4869,7 @@ private function configureFindBuilder(
         array $roles,
         PermissionType $forPermission,
         bool $qualifyCollidingGroups = true,
+        array $orderAttributes = [],
     ): bool {
         $hasSelectionProjection = false;
         if (! $hasAggregation) {
@@ -4839,14 +4888,20 @@ private function configureFindBuilder(
                     $selections,
                     includeInternal: ! $hasDistinct,
                     joinAliases: \array_column($joinTablePrefixes, 'alias'),
-                    joinSelections: $this->joinSelections($collection, $joinTablePrefixes),
+                    joinSelections: $this->joinWildcardSelections($collection, $joinTablePrefixes),
                 );
                 // The projection replaces the select; forwarded as well, the builder would compile the caller's
                 // raw attribute names whenever the projection holds only aliased joined columns.
                 $queries = \array_values(\array_filter($queries, static fn (BaseQuery $query): bool => $query->getMethod() !== Method::Select));
                 $hasSelectionProjection = true;
             } elseif (! empty($joinTablePrefixes)) {
-                $this->applyJoinProjection($builder, $collection, $joinTablePrefixes, $alias);
+                $this->applyJoinProjection(
+                    $builder,
+                    $collection,
+                    $joinTablePrefixes,
+                    $alias,
+                    $hasDistinct ? $selections : [...$selections, ...$orderAttributes],
+                );
                 $hasSelectionProjection = true;
             }
         }
@@ -6249,7 +6304,7 @@ private function remapRow(array &$row): void
             }
 
             $value = $row[$key];
-            if ($bare === Storage::PERMISSIONS || $public === Document::PERMISSIONS) {
+            if ($value !== null && ($bare === Storage::PERMISSIONS || $public === Document::PERMISSIONS)) {
                 $value = \json_decode(\is_string($value) ? $value : '[]', true);
             }
             if (! \array_key_exists($dotted, $row) || $key === $dotted) {

src/Database/Traits/Documents.php

@@ -5145,9 +5145,10 @@ private function joinedCollectionsByAlias(array $joins, ?array $joinedCollection
     }
 
     /**
-     * The `alias.$id` of each join whose attributes a select names without it, when an outer join
-     * can leave a joined row unmatched: the joined `$id` is what tells an unmatched row from a
-     * matched one when the row is decoded, so it is selected for that and left out of the result.
+     * The `alias.$id` of each join whose attributes a select names without it or `alias.*`, when an
+     * outer join can leave a joined row unmatched: the joined `$id` is what tells an unmatched row
+     * from a matched one when the row is decoded, so it is selected for that and left out of the
+     * result.
      *
      * @param  array<Query>  $selects
      * @param  array<Query>  $joins
@@ -5189,7 +5190,7 @@ private function outerJoinIdSelections(array $selects, array $joins, array $join
 
         $ids = [];
         foreach ($selectedAliases as $alias => $attributes) {
-            if (isset($joinedCollections[$alias]) && ! isset($attributes[Document::ID])) {
+            if (isset($joinedCollections[$alias]) && ! isset($attributes[Document::ID]) && ! isset($attributes['*'])) {
                 $ids[] = $alias.'.'.Document::ID;
             }
         }