Summary
A sandbox that is paused can no longer be resumed on a different node if the cross-node resume happens after the pause export's TTL (default 1 hour) expires.
When a sandbox is paused, the s3lvol side publishes a ref-layout export (lease-aware, pin=lease) so that a later cross-node resume can import it. If nothing imports it while the sandbox stays paused, the export becomes "terminal" once its lease is confirmed absent past the TTL deadline, and the reaper removes it automatically — manifest deleted, registry entry cleared. From that point on, any cross-node resume attempt fails because the manifest it must import returns 404. Same-node resume keeps working because it is served from the local pause catalog / lvstore snapshot volume and never depends on the export.
Environment
Description / mechanism
- Pause completes. The
s3lvol side publishes a ref-layout export for the pause package (manifest under <lvstore>/meta/exports/<uuid>.json) with lease_aware=true and the default TTL S3LVOL_EXPORT_DEFAULT_TTL_SEC = 3600 (1 h).
- Cross-node resume depends on that export:
restoreplace.Decide only allows a resume to leave the origin node while the pause record reports remote_status=ready (i.e. the export exists), and Cubelet on the destination node imports it through the export UUIDs (remoteUUIDs).
- If the export expires with no lease taken (
lease_absent — nothing imported it during the pause), export_lease_is_terminal() becomes true and the periodic exports_reap poller reaps it automatically: the manifest is deleted and the registry entry cleared. No explicit delete / release request is involved.
- Any later cross-node resume reads the already-removed manifest → 404 → the resume fails. Same-node resume is unaffected: it goes through the local pause catalog + lvstore snapshot volume and does not depend on the export.
Expected Behavior
A sandbox that is still paused should remain resumable on another node (cross-node) without being silently limited to the ~1 h export TTL. Reaching the export deadline should not invalidate a pause that is still owed to a possible cross-node recovery.
Actual Behavior
- The effective window for cross-node resume after a pause is the export TTL (default 1 h).
- Beyond that window, only same-node resume works. Cross-node recovery for the scenarios the export exists to serve — origin node failure / removal / drift — fails with a 404 on the export manifest.
Additional Context
Summary
A sandbox that is paused can no longer be resumed on a different node if the cross-node resume happens after the pause export's TTL (default 1 hour) expires.
When a sandbox is paused, the
s3lvolside publishes a ref-layout export (lease-aware,pin=lease) so that a later cross-node resume can import it. If nothing imports it while the sandbox stays paused, the export becomes "terminal" once its lease is confirmed absent past the TTL deadline, and the reaper removes it automatically — manifest deleted, registry entry cleared. From that point on, any cross-node resume attempt fails because the manifest it must import returns 404. Same-node resume keeps working because it is served from the local pause catalog / lvstore snapshot volume and never depends on the export.Environment
master— the automatic reaper behavior that causes this was introduced in s3lvol: pending-delete, derived exports, and faster same-bucket ingest #1663 (the fix for [Bug Report] s3lvol: an export whose lease is confirmed absent and whose TTL has lapsed is neither released nor stops polling, degrading into a 1 req/s 404 storm after restart #1634).Description / mechanism
s3lvolside publishes a ref-layout export for the pause package (manifest under<lvstore>/meta/exports/<uuid>.json) withlease_aware=trueand the default TTLS3LVOL_EXPORT_DEFAULT_TTL_SEC = 3600(1 h).restoreplace.Decideonly allows a resume to leave the origin node while the pause record reportsremote_status=ready(i.e. the export exists), and Cubelet on the destination node imports it through the export UUIDs (remoteUUIDs).lease_absent— nothing imported it during the pause),export_lease_is_terminal()becomes true and the periodicexports_reappoller reaps it automatically: the manifest is deleted and the registry entry cleared. No explicit delete / release request is involved.Expected Behavior
A sandbox that is still paused should remain resumable on another node (cross-node) without being silently limited to the ~1 h export TTL. Reaching the export deadline should not invalidate a pause that is still owed to a possible cross-node recovery.
Actual Behavior
Additional Context
rcow_get_exportson a paused sandbox shows the exports withlease_aware=true,lease_absent=true,pin=lease, andexpires_at= pause time + 1 h.CubeS3lvol/module/bdev/s3lvol/vbdev_s3lvol_exports.c(export_lease_is_terminal()+exports_reap()).CubeMaster/pkg/restoreplace/placement.go; same-node resume path:Cubelet/services/cubebox/pause_package.go.