Skip to content

[P2] rescore: truncated judge cache is permanently skipped instead of retried #364

Description

@reacher-z

Problem

The CLI's pending-selection predicate parses an existing judge file inside a broad try/except Exception: continue. A truncated JSON cache therefore skips the entire run before rescore_one gets a chance to recover it. The worker already handles invalid cached JSON, but is never scheduled in this case. Verdict files are written directly, so interruption can create this state.

Sources: pending selection, worker cache fallback and direct write.

Verified reproduction

  • Create a temporary intercepted run with valid run-meta.json and data/interception.json.
  • Write the incomplete text {"match": to judge.json.
  • Run the real rescore.main() with --only-batch, --rubric strict, a temporary dummy model configuration and a mocked judge.
  • Actual: exit 0, 0 judge calls, summary reports 1 judge error. A normal rerun repeats this behavior unless the user knows to use --force or delete the cache.

Acceptance criteria

  • Treat invalid JSON, non-object verdicts and invalid verdict types as unusable caches and schedule reevaluation with a diagnostic.
  • Share cache validation between the planner and worker so their recovery behavior cannot disagree.
  • Write verdicts atomically so interruption cannot replace a valid cache with partial JSON.
  • Test CLI-level scheduling, including both rubrics where one cache is corrupt and the other is valid.
  • Define/report partial-processing errors rather than quietly treating processing as wholly successful.

Suggested priority P2: recoverability and unattended batch reliability. #299 handled match: null; this is malformed JSON rejected before that recovery path. #303 concerned batch metadata, not judge cache writes. #313 does not change this predicate.

@Perry2004 please review recovery behavior and the appropriate partial-failure exit status.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions