fix(lima): reap orphaned host agents on stop - #50
Merged
Merged
Conversation
Lima can report an environment as stopped while stale hostagent processes continue consuming CPU. Converge stopped machines through the provider, terminate only exact Avar-owned hostagent matches, and escalate briefly when an agent does not exit cleanly.\n\nRegression tests cover the command path, stopped-state provider cleanup, and exact process matching.\n\nRequirements: REQ-5.2, REQ-5.4
A real-Lima E2E run showed that a spinning hostagent can keep limactl stop alive indefinitely, preventing the orphan cleanup from running. Bound graceful shutdown to 30 seconds, preserve caller cancellation, and escalate through the existing warned force-stop path when the internal deadline expires.\n\nRequirements: REQ-5.2
olamide226
added a commit
that referenced
this pull request
Sep 3, 2026
PR #50 made `avr stop` reap the orphaned Lima host agents a stopped instance can leave behind. The detector is right and was earned from a real leak; three defects sit around it, found by reading the merged code rather than by a failure. signalPID treats /bin/kill's exit status as the verdict, and killing a pid that has already exited exits 1. reapHostAgents returns that error and stopMachine wraps it into a failed stop, so an agent exiting on its own between the scan and the signal turns a successful stop into a reported failure. The escalation pass makes it likely rather than rare: it sends KILL to pids that received TERM 200ms earlier, which are exactly the ones exiting. avr stop is also the only command that passes types.DiscardProgress, at all four of its call sites, while every other command routes progress to stderr. The force-stop warning that unsaved work may be lost therefore reaches nobody. And idle auto-stop skips machines that are not running, which is the state a leaked agent leaves behind, so only an explicit avr stop ever self-heals. The task records the constraint on the obvious fix as well: this package compiles on Windows and hostagent.go carries no build tag, so syscall.Kill would reintroduce exactly what task 38 repaired. One item is deliberately written as a question rather than a finding — whether killing the agent takes an emulated instance's qemu process with it is unverified, and the task says to check it on a real machine before adding a predicate on suspicion. Requirements: 5.2, 5.5, 1.5
olamide226
added a commit
that referenced
this pull request
Sep 3, 2026
PR #50 made `avr stop` reap the orphaned Lima host agents a stopped instance can leave behind. The detector is right and was earned from a real leak; three defects sit around it, found by reading the merged code rather than by a failure. signalPID treats /bin/kill's exit status as the verdict, and killing a pid that has already exited exits 1. reapHostAgents returns that error and stopMachine wraps it into a failed stop, so an agent exiting on its own between the scan and the signal turns a successful stop into a reported failure. The escalation pass makes it likely rather than rare: it sends KILL to pids that received TERM 200ms earlier, which are exactly the ones exiting. avr stop is also the only command that passes types.DiscardProgress, at all four of its call sites, while every other command routes progress to stderr. The force-stop warning that unsaved work may be lost therefore reaches nobody. And idle auto-stop skips machines that are not running, which is the state a leaked agent leaves behind, so only an explicit avr stop ever self-heals. The task records the constraint on the obvious fix as well: this package compiles on Windows and hostagent.go carries no build tag, so syscall.Kill would reintroduce exactly what task 38 repaired. One item is deliberately written as a question rather than a finding — whether killing the agent takes an emulated instance's qemu process with it is unverified, and the task says to check it on a real machine before adding a predicate on suspicion. Requirements: 5.2, 5.5, 1.5
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the lifecycle gap where Lima reports an Avar environment as stopped while orphaned
limactl hostagentprocesses continue consuming CPU. Bothavr stopandavr stop --allnow converge already-stopped environments through the provider, which removes only exact host-agent matches for the owned machine and uses a bounded TERM-to-KILL escalation for wedged agents.A real-Lima E2E run also exposed that a spinning host agent can keep
limactl stopitself alive indefinitely, preventing orphan cleanup from running. Graceful shutdown is now bounded at 30 seconds before Avar enters its existing warned force-stop path.Regression coverage locks in the previously missed command path, the stopped-state provider cleanup call, exact process matching, and the hung graceful-shutdown path.
Spec task: tasks.md task 10 — Implement
avr statusandavr stop(corrective bug fix)Requirements covered
avr stop, stop the selected machine;avr stop --allstops all Avar-managed machines.limactlexecutable plus the exact Avar machine argument.Correctness properties
TestParseOrphanHostAgentPIDs_MatchesOnlyTheExactLimaMachine, existing provider ownership tests, andTestStop_AllStopsEveryOwnedMachineAndNothingElse_REQ_5_2Deliberately not covered
None. This is a corrective lifecycle fix for the already-completed stop task; no
tasks.mdcheckbox changes are needed.Verification
make lintmake tidy-checkmake release-version-testmake buildmake testmake e2e— passed against real Lima on the affected Mac:ok github.com/olamide226/avar/e2e 390.719s./bin/avr stop --all— stopped three running test environments; all six Lima environments reportStoppedand no Avarlimactl hostagentprocess remains.git diff --checkNotes for review
The first real-Lima E2E attempt reproduced a second part of the failure:
limactl stop avr-ubuntu-24.04-arm64remained alive for more than four minutes while a child host agent consumed roughly 95% CPU. That run was interrupted, the three test VMs were force-stopped directly through Lima, and commit2750179added the bounded graceful-stop path plusTestStop_EndsAMachineWhenGracefulShutdownHangs_REQ_5_2. The complete E2E rerun then passed.Lima 2.2.0 can lose track of stale host agents once its pid/socket files have been replaced or removed; another
limactl stop --forcethen reports the agent as already stopped and does not reap those processes. The cleanup therefore inspects the macOS process table after provider stop convergence. It sends TERM first, waits 200 ms, and uses KILL only for matching agents that remain. Orphan cleanup is bounded by two seconds.