Production Benchmark Qualification & Reliability Hardening

[wangxingjun778]

Features

• Benchmark qualification suite (leap-bench): New standalone package — manifests, registry, async runner, unified metrics, JSONL evidence, and ready/conditional/blocked gates over tier0–tier4 (plus live-llm, production-sim, pre-hardware profiles), with a leap-bench CLI (list/doctor/run/resume/report/gate) and a benchmark install extra.
• 36 native scenarios across 11 adapters: Refusal, approval & trust, safety policy, degradation, side-effect uncertainty, control jitter, fleet isolation, PCD efficiency, YAML onboarding, and governance — all against public APIs.
• 12 external integrations with strict official contracts: ASIMOV, AttackVLA, CALVIN, EmbodyGuard, IS-Bench, Isaac Lab, KinDER, ManiSkill, RoboJailBench, Safety-Gymnasium, VLABench, and an opt-in live-LLM harness; each returns typed UNAVAILABLE until an explicit local command and data root are configured — no auto-download, no fabricated results.
• Zero-motion Tier-4 hardware qualification with durable evidence: Preflight requires preflight_mode: zero_motion, per-device required checks, and a structured report asserting motion_performed=false; any motion claim or missing evidence fails closed. Every run is isolated under benchmarks/runs/<run_id>/<adapter>/, replacing temp output.
• Governed, fail-closed hardware degradation: New DegradationCoordinator is the single execution point for declared policies (halt/hold_position/safe_return/switch_sensor/continue_blind) across stream and health-monitor paths; latched states block writes until verified recovery or manual clear, with new home_joint_targets, sensor_fallbacks, and bounded safe_return_timeout_s.
• Task-completion evidence and terminal failure fingerprints: A per-frame artifact ledger forces final answers once requested artifacts are confirmed; FailureFingerprint normalizes volatile output so repeated identical failures terminate instead of being retried blindly.
• CUA window-target contract: Window screenshots require a validated pid+window_id, malformed discovery is rejected, and bounded recovery plus a dependency circuit stop repeated RPCs on a known-bad driver — with MCP calls marshaled onto the loop that owns the stdio session.
• Explicit slash-command routing and /host actions: Free text is only parsed as a command with a leading slash; /host declares actions once for usage, completion, and both runtimes, reporting driver health, failures, and recovery steps.

Enhancements

• Failure visibility end to end: TUI renders a stable state prefix (code, retry, repeat, blocked, cooldown, exit) with next-step suggestions and tail-preserved tracebacks; shell_run returns structured failure codes with non-sensitive audit metadata; recovery classifies producer-declared terminal codes into a new non-recoverable tool_terminal category before text heuristics.
• Adaptive context retention: Working memory auto-sizes from the active model window (1/8, bounded 16K–128K) instead of the legacy 8K buffer; expanded turns keep 12 recent messages and compressed summaries retain tool evidence.
• CI & packaging: Ruff and the benchmark contract suite now cover src/benchmarks/tests/benchmarks; Tier0/Tier1 smoke runs wired into CI; manifests ship as package data; 14 new benchmark test modules plus coverage for slash routing, tool hardening, hardware degradation, and CUA mapping.

Fixes

• Daemon lifecycle: Non-child zombie processes are detected, so a stopped daemon no longer reports as alive.
• Evidence across long turns: Tool results and completion state survive tool-heavy turns and context compaction.
• Command misrouting: Natural-language input beginning with a command word no longer executes a command.

Refactor

• Degradation execution unified into DegradationCoordinator (no more forked halt/hold/safe-return logic); repetition guard switched to normalized structured fingerprints.