fix(controllers): reconcile not-Ready pods against their placeholder job#37
Open
cgetzen wants to merge 1 commit into
Open
fix(controllers): reconcile not-Ready pods against their placeholder job#37cgetzen wants to merge 1 commit into
cgetzen wants to merge 1 commit into
Conversation
The pod controller only reconciled pods that were Running and Ready, so
a pod that never became Ready (crash-looping, stuck in Init) was skipped
on every reconcile. When Slurm preempted or cancelled its placeholder
job, the pod leaked indefinitely while Slurm reissued the freed GRES to
the next job. The scheduler then stamped a second ResourceClaim for the
same device, and the successor pod wedged in Init on
NodePrepareResources ("requested device is already allocated to
different claim").
Gate reconciliation on binding instead of readiness: a bound pod may
exist iff its placeholder job is RUNNING. The scheduler binds a pod only
after its job starts, so RUNNING is the only healthy post-bind job state
- a bound pod whose job is terminal, gone, requeued, or suspended is
deleted regardless of readiness, releasing its resources on the
Kubernetes side before Slurm hands them to another job. Unbound pods are
never reconciled; the scheduler owns the pre-bind lifecycle.
Signed-off-by: Charlie Getzen <charliegetzen@meshy.ai>
cgetzen
requested review from
SkylerMalinowski,
alanmutsch,
catblade,
vivian-hafener and
wickberg
as code owners
Contributor
|
Thanks for reporting and fixing this @cgetzen - I'll take a look. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #36.
The pod controller only reconciles a pod against its Slurm placeholder job while the pod is
RunningandReady. A pod that never becomes Ready (crash-looping container, stuck in Init, failing readiness probe) is skipped on every reconcile. When Slurm preempts or cancels its placeholder job, the pod persists indefinitely. On DRA clusters the stale pod retains itsResourceClaimdevice allocation while Slurm reissues the freed GRES to the next job; the scheduler then stamps a second claim for the same device and the successor pod wedges inInitonNodePrepareResources("requested device is already allocated to different claim"), requiring manual intervention.This PR gates reconciliation on binding instead of readiness: A bound pod may exist only if its placeholder job is
RUNNING. This is sound because the scheduler only binds a pod after its job starts, so post-bind,RUNNINGis the only healthy job state.Checklist
CONTRIBUTING.md
and the
Code of Conduct.
Breaking Changes
No API changes. Exactly one behavioral delta: bound pods that are not Ready are now reconciled — a crash-looping (or still-starting) pod whose placeholder job stops running is deleted instead of leaking forever. This is the fix. Bound Ready pods, unbound pods, and unlabeled pods have no behavior changes.
Testing Notes
go test ./internal/controller/...— ginkgo cases covering every combination: crash-looping (job running vs job gone), queued (job pending, job gone, unlabeled), bound-requeued, suspended, and bound-unlabeled orphan pods.hack/kind.sh --core): bridge-scheduled crash-looping pod (never Ready) + healthy control pod;scancelthe crash-looper's placeholder job, results in pod deletion within one 30s reconcile (Deleting Pod for corresponding Slurm Job), claim released, healthy pod untouched.