fix(orchestrator): restore pause state after allocation failure

[morluto]

What does this PR do?

Restores a running sandbox when paused-artifact-root allocation fails before the backend pause begins.

Previously, pause_sandbox_inner detached the runtime handle and proxy route and then propagated an allocation error with ?. That skipped the rollback used by later pause failure paths, leaving metadata in Pausing while the live runtime was no longer routable.

The failure path now restores the runtime handle and proxy route, transitions metadata from Pausing back to Running, and releases the temporary PausedSandbox image ownership before returning the allocation error. If the metadata transition also fails, the returned internal error preserves both failures.

                         allocate artifact root
Running -> Pausing -> detach handle + route --------X
                                                     |
                                                     v
                  release paused refs <- Running <- restore route + handle
                                                     |
                                                     v
                                          return allocation error

Closes #99.

How was it tested?

• Added pause_artifact_root_allocation_failure_restores_running_for_retry, which deterministically fails the first artifact-root allocation and verifies:
  • metadata returns to Running with no paused state;
  • the runtime handle and ready proxy route are restored;
  • temporary paused-image ownership is released;
  • no paused state is persisted for the failed attempt; and
  • a second pause succeeds and reaches Paused.
• cargo test -p agentenv --lib orchestrator::service::tests::pause_artifact_root_allocation_failure_restores_running_for_retry -- --exact --nocapture
• cargo fmt --all -- --check
• git diff --check main...HEAD

Compatibility

No API, persistence format, or successful pause behavior changes. This only changes rollback after an artifact-root allocation error that occurs before SandboxBackend::pause is called.

Suggested review order

1. Review the allocation error branch in src/orchestrator/service.rs.
2. Review the deterministic failure and retry assertions in src/orchestrator/tests.rs.

[github-actions]

🔍 OpenCodeReview found 3 issue(s) in this PR.

• ✅ Successfully posted inline: 3 comment(s)

[github-actions]

[bug · medium]
This discards the persistence deletion failure after the metadata has already been removed, leaving no retry path. A stale paused record is loaded back into the metadata store by load_all on restart, so this can resurrect the sandbox that this branch just removed (and orphan artifacts in the meantime). Make failed deletion durable/retryable via reconciliation, or preserve enough metadata to retry cleanup rather than reporting only SandboxNotFound.

[github-actions]

[bug · high]
The runtime and proxy route become active before the fallible metadata rollback. update_state_if_state returns an error on a backend failure or state conflict; in that case this branch returns with a routable runtime while metadata may remain Pausing. Concurrent pause/delete calls can then wait indefinitely for that transition, while data-plane traffic still reaches the sandbox. If publishing Running fails, undo the handle/route restoration and stop or otherwise reconcile the runtime before returning; add a failure-injection test for this path.

[github-actions]

[test · high]
This expectation legitimizes releasing the only durable hold on runtime_artifacts as soon as the sandbox is restored. A maintenance pass can snapshot the running set while the handle is detached, then continue after this unpin; its stale snapshot contains neither this sandbox nor a hold, so GC may delete artifacts still used by the restored runtime. Please change the rollback implementation and this assertion so protection is retained or atomically transferred to a running/runtime owner until maintenance is guaranteed to observe the restored handle, and add a test that interleaves maintenance with this rollback.