fix(deploy): let a catch-up migration outlive the 10-minute Job deadline - #270
Merged
Merged
Conversation
The candidate migration Job runs with activeDeadlineSeconds=600 and a poller of 200 attempts x 3s, both hardcoded. That is sized for an ordinary deploy, which applies at most one small migration. A catch-up deploy is a different shape: with many migrations pending it must apply all of them in one Job, including a CREATE INDEX CONCURRENTLY over a production-sized table, and 10 minutes is not enough. Kubernetes then kills the pod with DeadlineExceeded, which surfaces as a bare job_failed, and the pod is deleted with its logs. The deadline is now a workflow input defaulting to the same 600s, validated as an integer in [60, 2700] before it can reach activeDeadlineSeconds -- a non-numeric value would otherwise fall through to the script's own 600s default and silently discard what the operator asked for. Poll attempts are derived from it with a 20-attempt margin so the poller outlives the Job and reports the real terminal condition instead of job_timeout. The job budget goes to 75 minutes to cover the new ceiling plus rollout and both smokes. The default is deliberately unchanged: a tight deadline on a normal deploy is what makes DeadlineExceeded mean 'stuck' rather than 'busy'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deploy run 34529081984 failed with
deploy-release: job_failedafter 10m14s. The Job's own events say why:activeDeadlineSecondsis600, and the poller is200 × 3s— also 600s — so both bounds land together and the failure arrives as a barejob_failedwith the pod already deleted and its logs gone.600s is the right number for an ordinary deploy, which applies at most one small migration. It is the wrong number for a catch-up deploy that has to apply every pending migration in a single Job, one of which is a
CREATE INDEX CONCURRENTLYover a production-sized table.What changes
migration_deadline_secondsbecomes a workflow input, defaulting to the same 600. Ordinary deploys are unaffected.[60, 2700]in the resolve step. Without that check a non-numeric value reachesNumber(...)indeploy-release.mjs, becomesNaN, and falls through to that script's own 600s default — silently discarding what the operator asked for.deadline / 3 + 20, so the poller outlives the Job's deadline and reports the terminal condition Kubernetes actually recorded rather thanjob_timeout.timeout-minutes45 → 75, covering the 2700s ceiling plus the 600s rollout and both smokes.Why the default stays at 600: a deadline sized for the common case is what makes
DeadlineExceededmean stuck rather than busy. Raising it globally would remove that signal from every deploy to serve a one-time backfill.On safety of the kill that prompted this: migration 0002 is
transactional: true, so the killed pod rolled back rather than leaving a partial detach; 0005 and 0006 aretransactional: falsebut useensureConcurrentIndex, which repairs an INVALID index left by an interrupted build. The Deployment was never patched — the migration is the step before it. Production is on its previous image and the schema is intact and rerunnable, which is the behaviourdocs/RELEASE.mdalready promises.The residual Job is terminal (
Failed=True,activeunset) and carries no active writer, solistResidualMigrationJobswill not block the retry.