Skip to content

Preserve native payload storage across write-policy changes - #1224

Merged
rmcdaniel merged 9 commits into
mainfrom
fix/546-retained-storage-policy
Nov 15, 2026
Merged

rmcdaniel merged 9 commits into
mainfrom
fix/546-retained-storage-policy

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Disabling uploads or moving a namespace's write policy currently strands previously offloaded payloads. This native local-storage fix keeps each object's original namespace, storage generation and canonical root, so retained reads, run pruning, abandoned-upload cleanup and namespace deletion continue against the original location. New uploads must still pass the current enabled policy; a write already admitted may finish at its recorded location.

New schema steps add an internal binding and policy generation on SQLite, PostgreSQL, MySQL and MariaDB while preserving every historical bootstrap checksum. Older registrations are bound only when an admitted policy reproduces their recorded URI, including before a policy replacement. Missing or substituted roots refuse without creating replacement directories. Per-object deletion journals preserve mixed generations, backup holds and other namespaces' ownership of the same physical object. The public reference shape is unchanged.

Local validation: 545 library tests passed (98 declared backend/subprocess ignores); all 51 external-payload HTTP cases passed on SQLite, MySQL 8.0, MariaDB 10.11 and PostgreSQL 17. Seventeen backend upgrade/takeover cases passed, including both MySQL/MariaDB interrupted ALTER states, frozen-layout takeover and pending deletion journals. The physical-schema comparators validate both native columns exactly before comparing every remaining PHP column, index and constraint; two local source tests passed 97 assertions including changed-type/default/nullability, duplicate/missing-column and unrelated-drift controls. All 63 current-head checks, merge-queue checks and all four actual-main workflows passed. The merge tree exactly equals qualified head 1c4fecd1c2dc6923ccd6767d9bdb12285bd1fc14; accepted main is e363a472dd9183a66d31e1554824eaceb595253b. No published-artifact or release qualification is claimed.

Fixes #546.

The parity contract preserves published PHP’s disabled-policy refusal and requires native retention to delete through its original binding, then report run_not_found for the already pruned run. The two retention steps and three namespace-deletion steps, plus the expected 404 when re-enabling that already deleted namespace, are the only native expectation changes; frozen PHP setup and step values are unchanged. The focused 39 retention, namespace-deletion and native-divergence controls pass; earlier discovery controls also pass. The new schema tests are explicitly selected by the required PHPUnit workflow (9 local tests / 230 assertions), and its existing exact Composer command inventory is rebound without changing execution guards. The final hosted checks passed, including the corrected native retention and namespace-deletion expectations. Topic refs are absent and the worktree is clean.

@rmcdaniel rmcdaniel left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security contract review HOLD on exact test-only head 9d763ba244de3e358064f2cda5d3ecd93508ccda.

The disable/read/prune and disabled-namespace-deletion regressions point in the right direction, but the location-change test encodes the wrong settled product boundary. It requires 409 external_payload_storage_in_use until every existing registration is released. The later owning decision on #546 is explicit: already-stored references keep their original location, while disable/move/prefix changes apply to new writes. A retained object therefore must not pin the namespace's write policy or prevent an administrator from moving future writes.

Before production implementation, replace that refusal contract with a causal two-generation test:

  • upload and retain object A under the original local binding;
  • successfully move the namespace policy while A still exists;
  • verify A remains readable from the original binding;
  • verify a new object B is written only to the replacement binding;
  • prune/delete A through its original binding without touching B, then prove both registrations and objects reconcile;
  • cover namespace deletion when retained objects span both bindings.

The source contract also still needs the issue's remaining security/error seams:

  • an original binding that can no longer be resolved must return a specific typed reason, not external_payload_unavailable / “temporarily unavailable”;
  • preserve namespace ownership and refuse cross-namespace reference/policy substitution;
  • bind the storage policy/version needed for each registration rather than deriving old-object authority from the mutable current namespace policy. Do not copy credential values into object rows; retain only the provider/location identity or credential reference needed by the supported driver;
  • add a concurrent policy-change/upload control proving a write is wholly registered under exactly one immutable policy generation, and that disabling still prevents every new offload.

The current runtime_external_payloads registration records the URI but not a policy generation, while fetch and retention currently construct the driver from the mutable namespace policy. The implementation must close both reads and deletion/reclamation for the original generation. This review does not select a schema or implementation, but the tests must express the settled behavior before they can serve as acceptance evidence.

@rmcdaniel

Copy link
Copy Markdown
Member Author

One additional implementation requirement before the schema shape freezes: existing registrations and policy-update races need an explicit binding transition, not only the new-write path.

  • SQLite, MySQL and PostgreSQL upgrade paths must bind every existing runtime_external_payloads row to the namespace policy/location that exists during the migration. A retained pre-upgrade row must then remain readable and deletable from that binding after the namespace moves to a new location or disables new writes.
  • A missing, malformed or unsupported policy during backfill cannot be silently resolved through whatever policy becomes current later. Preserve the row and return the new typed unresolvable-original-binding reason. Add an upgrade fixture covering a ready row at location A, backfill, change to B, read/delete A, and create B-only new data; include the missing/invalid-policy refusal.
  • Upload admission and set_external_policy need one namespace-row linearization point on every upload, not only when quota locking happens. The policy/location used to construct the provider, the registration binding stored before the provider effect, and the generation observed by the update must agree. A deterministic race control should prove an upload is wholly generation A or wholly generation B, never an A object with a B binding or vice versa.
  • Keep the binding namespace-scoped even when two namespaces resolve to the same object URI. Do not borrow another namespace’s generation or copy future provider credential values into the registration.

These are source/upgrade requirements only; the existing contract HOLD remains until the corrected tests and implementation are concrete.

@rmcdaniel

Copy link
Copy Markdown
Member Author

One more consequence of allowing location changes while retained objects remain: cleanup plans must be per-object binding, not one namespace policy per prune.

The current retention_payloads::Plan carries one policy for every target, and namespace cleanup similarly starts from the current namespace policy. After a permitted A→B change, one run or namespace can legitimately contain A-bound and B-bound registrations at the same time. Reading may work while pruning or namespace deletion still sends both URIs through B, deletes the wrong path, or strands A.

Please make each target carry its own immutable binding and stage each dw_external_payload_deletions row from that binding. Add a mixed-generation control where one retained run references an A object and a B object, then prune/delete the namespace and prove each object is deleted only through its own root; failure of one generation must retain/refuse that target without borrowing the other generation. Include the same-URI/cross-namespace case so deduplication cannot substitute another namespace’s binding.

This is part of the existing contract HOLD, not a new product scope.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The same per-binding correction must include expired unclaimed cleanup. payload_cleanup_pass currently receives one current LocalStorage, and cleanup_reference merely checks whether that current root reproduces the row URI. After A→B, an expired A registration would remain permanently blocked even though its original binding is valid.

Resolve the storage binding from each candidate row, preserve the object/backup-hold locks, and add an A+B cleanup pass proving both expired objects are reclaimed through their own roots. Missing original binding should increment the bounded failure/blocked accounting with the typed reason; it must not fall back to B.

@rmcdaniel rmcdaniel changed the title Preserve retained native payloads across storage policy changes Preserve native payload storage across write-policy changes Nov 15, 2026
@rmcdaniel
rmcdaniel marked this pull request as ready for review November 15, 2026 05:13
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit e363a47 Nov 15, 2026
64 checks passed
@rmcdaniel
rmcdaniel deleted the fix/546-retained-storage-policy branch November 15, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Keep stored external payloads readable after a namespace storage policy is disabled or moved

1 participant