Skip to content

fix: keep host-lifecycle summarization failures quiet and isolate test logs - #1245

Merged
jalehman merged 1 commit into
mainfrom
fix/stale-runtime-llm
Oct 11, 2026
Merged

jalehman merged 1 commit into
mainfrom
fix/stale-runtime-llm

Conversation

@jalehman

@jalehman jalehman commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

What

When a summarizer call fails because the OpenClaw host runtime retired (Plugin inventory has retired; begin a new plugin operation. or Async work scope is closed), Lossless now logs one warning instead of two error lines. The pending summaries and compaction debt stay queued for the next drain, as before. The vitest suite also stops writing to the production /tmp/openclaw/lossless-claw-<date>.log.

Why

The root cause of the retired-inventory failure is in the host, not in Lossless. The context engine replays a stale async-context snapshot that was captured when its factory resources were created. api.runtime.llm then resolves model metadata against the retired plugin cache. openclaw/openclaw#168916 fixes this (it closes openclaw/openclaw#156026). For related background on Lossless's handling of host lifecycle errors, see the closed #1195.

On the Lossless side, two things made this failure hard to triage.

First, each interrupted summarization was logged as two error lines: [lcm] runtime.llm.complete error: … and the bare message. Neither line said that the work was kept. Live evidence from 2026-10-08 (sanitized):

  • 08:44:41 PDT: [reload] plugin metadata changed with identical config; applying plugin lifecycle → Plugin replacement applied: knowledge. This was an unrelated plugin.
  • 54 s later: [lcm] runtime.llm.complete error: Plugin inventory has retired; begin a new plugin operation. → maintain: deferred compaction skipped ... reason=Plugin inventory has retired → background deferred compaction done ... changed=false. The debt was retained and then waited under deferred compaction backoff active.

Second, local vitest runs wrote to the same production rolling log. On 2026-10-08 that log held about 700 synthetic errors from tests: 624 "inventory retired", 39 "model override not allowlisted", and 39 "Async work scope is closed". These were mixed in with the real event above.

Changes

  • The pending-compaction boundary logs one warning that names the cause and says the work is kept
  • runtime.llm.complete host-lifecycle rejections are classified without an error log
  • The summarizer rethrows lifecycle errors without logging; other errors are unchanged
  • The coordinator marks interrupted (claim-released) failures as lifecycleInterrupted
  • vitest sets LCM_LOG_FILE to a per-run temp file, with a sandbox assertion
  • Two plugin-config log tests clear the env override for themselves

The retained-work behavior is unchanged. The pending worker already releases the node claim untouched, and the debt row stays pending under the existing backoff. No retry loops or timers were added.

Testing

  • npm run typecheck
  • npm run build
  • npm test (run twice): 135 files and 2122 tests passed both times
  • A new parameterized test in test/index-runtime-llm-complete.test.ts covers both host-lifecycle messages. It checks that a background maintain() keeps the debt pending, leaves the nodes planned with retryCount=0 and no lease, writes no summaries, logs exactly one warning, and logs no error. The test fails on main.
  • Production log check: the log grew during the runs only because the live Gateway was writing to it. No line written during the run windows matched test markers (vitest, the test session IDs, /tmp/lossless-claw-* DB paths, or the lifecycle and allowlist messages). The test logs land in <vitest HOME>/logs/lossless-claw-test.log.

…t logs

Host runtime retirement ("Plugin inventory has retired", "Async work
scope is closed") was logged as two error lines per summarizer call. The
summarizer now rethrows it silently and the pending-compaction boundary
logs one warning naming the cause; the existing retained-work behavior is
unchanged. vitest sets LCM_LOG_FILE to a per-run temp file so test runs
never append to the production /tmp/openclaw rolling log.
@jalehman
jalehman force-pushed the fix/stale-runtime-llm branch from 55fe011 to f5b60ac Compare October 11, 2026 05:48
@jalehman jalehman changed the title fix: route background summarization through the newest live registration fix: keep host-lifecycle summarization failures quiet and isolate test logs Oct 11, 2026
@jalehman
jalehman merged commit 6a5915f into main Oct 11, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant