Skip to content

feat(cosh-ng): add diagnostics and health troubleshooting - #3312

Draft
zkl123 wants to merge 8 commits into
agentic-os-org:mainfrom
zkl123:feature/cosh-ng/diagnostics
Draft

zkl123 wants to merge 8 commits into
agentic-os-org:mainfrom
zkl123:feature/cosh-ng/diagnostics

Conversation

@zkl123

@zkl123 zkl123 commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

This PR introduces a comprehensive diagnostics and health troubleshooting experience for cosh-ng.

What changed:

  • Redesign the cosh doctor CLI output with structured runtime and log evidence.
  • Probe the live cosh-core process from the /health endpoint.
  • Register running shell and core processes for diagnostics correlation.
  • Persist panics to crash files for post-mortem analysis.
  • Dual-write key adapter errors to logs for better observability.
  • Default log level to info to reduce noise while keeping actionable events.
  • Add a troubleshooting user guide.

Validation:

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace

no-issue: internal diagnostics improvement

Kailong Zhou added 8 commits September 15, 2026 20:25
The warn default filters out nearly every runtime event; info surfaces
process/session lifecycle, adapter transport and approval nodes while
keeping the COSH_LOG > RUST_LOG > TOML priority chain unchanged.

Signed-off-by: Kailong Zhou <zhoukailong.zkl@alibaba-inc.com>
Assisted-by: Qoder <noreply@qoder.io>
Errors converging on the service loop, registry transport, recovery
state transitions and the core output thread previously reached only
the UI event channel, leaving logs empty when cosh-shell quit silently.
Record them at the choke-points and document the info event checklist
and warn/error dual-write policy in AGENTS.md.

Signed-off-by: Kailong Zhou <zhoukailong.zkl@alibaba-inc.com>
Assisted-by: Qoder <noreply@qoder.io>
Panics only reached stderr, which the TUI consumes, so crashes left no
durable evidence. Append one JSON line per panic to
~/.copilot-shell/cosh-{shell,core}-crash.log from the panic hooks and
chain the previous hook; the doctor crashes collector reads the file.

Signed-off-by: Kailong Zhou <zhoukailong.zkl@alibaba-inc.com>
Assisted-by: Qoder <noreply@qoder.io>
Write atomic <kind>-<pid>.json entries under ~/.copilot-shell/run/ so
doctor can pair shell and core offline and detect leftovers from
abnormal exits. Shells record routing facts and the spawned core pid;
cores record the owner shell pid via COSH_SHELL_PID. Entries are
removed on clean shutdown and stale dead-pid entries older than 7 days
are cleaned at startup.

Signed-off-by: Kailong Zhou <zhoukailong.zkl@alibaba-inc.com>
Assisted-by: Qoder <noreply@qoder.io>
Add two environment collectors: runtime (live shell/core pairing,
stale run-registry entries, 24h crash records as critical findings)
and logs (bounded tail scan of the daily log files for 24h WARN/ERROR
counts, a redacted error sample, and write-freshness facts). Doctor,
/health and diagnostics export share the findings through
run_env_collectors; crash records exit 2, other runtime anomalies warn.

Signed-off-by: Kailong Zhou <zhoukailong.zkl@alibaba-inc.com>
Assisted-by: Qoder <noreply@qoder.io>
Extend the /health slash command with a live block that only a running
session can observe: core liveness via a registry query that never
falls back to a short-lived child, session recovery state, and shell
routing facts (AI/assistance/integration, marker generation, zsh
command-not-found handler ownership, last route decision). A live core
combined with routing facts that explain a fallback is reported as a
named finding — routing compatibility fallback, not a provider failure.

The live-registry timeout path now releases the busy flag so a core
that dies mid-turn cannot force probes onto the short-child path and
report the dead core as healthy. Routing facts reach exports offline
through the run-registry snapshot; transient live probes stay out.

Signed-off-by: Kailong Zhou <zhoukailong.zkl@alibaba-inc.com>
Assisted-by: Qoder <noreply@qoder.io>
Rework `cosh-shell doctor` from a bare findings list into a readable
diagnostic report. The render layer now assembles a static fact header
between the status line and the checks list: version and host from the
binary, runtime and routing from the on-disk run registry (live
pid-alive entries, AI/integration/command-not-found-handler/last-route
facts), log level plus 24h ERROR counts and freshness from the daily
logs, and a 24h crash count with timestamps. The machine contract is
unchanged (`status:`/`checks:` lines, `[warning]`/`[critical]`/
`[unavailable]` labels, exit codes 0/1/2), and the header is pure local
reads so bundle/doctor consistency tests stay valid.

When findings or unavailable checks exist, a final line points at
`cosh-shell diagnostics export` for evidence collection; `/help` gains
a one-line diagnostics hint (in-session /health first, then doctor).
Eleven new MessageIds carry the en/zh wording.

Signed-off-by: Kailong Zhou <zhoukailong.zkl@alibaba-inc.com>
Assisted-by: Qoder <noreply@qoder.io>
Ship the bilingual user troubleshooting guide (en/zh) that turns the new
diagnostics evidence into a procedure. It organizes the failure classes
cosh-ng can suffer silently — a shell exit without an error, leftover
cosh-core processes, and input that stops routing to the Agent — into a
three-step SOP: run /health or cosh-shell doctor, read the on-disk
evidence (~/.copilot-shell logs, crash logs, run registry), and export a
redacted bundle with --since-hours when the incident predates the 24h
window. A zsh input-routing matrix carries the issue agentic-os-org#3055 differential
probe (?? versus bare natural-language input, _COSH_AI_ENABLED and
_COSH_HAS_USER_COMMAND_NOT_FOUND) and its interpretation branches.

The doctor export hint and the /help diagnostics line now carry the
guide path, and both README indexes link the new page.

Signed-off-by: Kailong Zhou <zhoukailong.zkl@alibaba-inc.com>
Assisted-by: Qoder <noreply@qoder.io>
@github-actions github-actions Bot added component:cosh-ng src/cosh-ng scope:documentation ./docs/|./*.md|./NOTICE labels Sep 17, 2026
@zkl123
zkl123 marked this pull request as draft September 17, 2026 03:26
@SunnyQjm

SunnyQjm commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

PR number: #3312
head_sha: bd55072
reviewed_at: 2026-09-17T04:02:18Z

Findings

  • [P1] src/cosh-ng/crates/cosh-shell/src/i18n/message_id/health.rs:98 — 破坏稳定 MessageId 判别值。 新 ID 插入既有分段,使后续公开变体整体偏移 41;i18n/mod.rs:114 修改固定断言反而掩盖了兼容性回归。新分段必须追加到所有既有 ID 之后,并保持原 ordinal。

  • [P1] src/cosh-ng/crates/cosh-shell/src/runtime/terminal.rs:84 — 信号处理器执行非 async-signal-safe 操作。 remove_shell() 涉及 OnceLock、互斥锁及 Rust 文件系统代码,信号中断相关路径时可能死锁或产生未定义行为。处理器应只设置原子标志,或保留 registry 作为异常退出证据。

  • [P1] src/cosh-ng/crates/cosh-shell/src/diagnostics/run_registry.rs:308 — 仅用 PID 存活判定进程身份。 遗留条目的 PID 被无关进程复用后,doctor 会把该进程误判为 orphan core 并建议执行 kill {pid}。需校验启动时间、可执行文件或其他不可复用身份。

  • [P1] src/cosh-ng/crates/cosh-shell/src/diagnostics/health/env_collectors.rs:698 — 向已有约 698 行 production 文件新增约 493 行,估算达到约 1191 行。 这越过 1000 行阻断线且未登记,应将 runtime/crash 与 logs collector 拆到独立 owner 子模块;最终行数以 layout gate 实测为准。

  • [P2] src/cosh-ng/crates/cosh-shell/src/shell_host/osc/routing.rs:30 — shell_host 直接修改 diagnostics 全局状态并触发磁盘写入。 adapter/cosh_core_service.rs:405 也存在同类反向依赖。底层 parser/driver 应产出结构化事件,由 runtime/diagnostics owner 持久化。

  • [P2] src/cosh-ng/crates/cosh-shell/src/diagnostics/health/env_collectors.rs:1045 — “24h ERROR/WARN 数量”实际只扫描每个日志末尾 256 KiB。 默认日志级别升至 info 后,高流量日志很容易截掉仍处于 24 小时窗口内的错误,导致 doctor 静默少报;应流式扫描时间窗口或明确标为尾部样本。

  • [P2] src/cosh-ng/crates/cosh-shell/src/diagnostics/crash.rs:82 — crash log 截断可能在 panic hook 内再次 panic。 existing[keep_start..] 使用任意字节偏移切割 UTF-8;包含非 ASCII panic/path 且日志越过上限时可能落在字符中间。cosh-core/src/crash.rs 有相同问题,应先对齐字符边界。

Validation

未执行本地验证。当前状态显示完整 Test cosh-ng 成功,但 Test cosh-ng fast checks 仍失败。

@SunnyQjm

SunnyQjm commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

PR number: #3312
head_sha: bd55072
verified_at: 2026-09-17T04:16:16Z
tier: T1

1. 验证范围

  • HEAD 校验通过。
  • Backend:Apple container
  • RUN_ID:20260917T040527Z-180ecec1
  • 已运行:
    • cargo test -p cosh-shell --lib
    • cargo test -p cosh-shell --test raw_cli diagnostics -- --test-threads=1
    • cargo test -p cosh-core
  • 宿主 doctor 初次报告 contended=true;不阻塞正确性验证。
  • test plan 因本地 origin/main 基线过旧包含大量无关变更,故保持作业指定 scope。

2. 结果分类

失败

Scope 首跑 复跑
cosh-shell lib 1502 passed,0 failed 未复跑
raw_cli diagnostics 3 passed,0 failed 未复跑
cosh-core 979 passed,20 failed 979 passed,20 failed

cosh-core 两次均以退出码 101 失败,失败集合一致,涉及 hook、MCP 和 session store 测试,不属于 suspected-flaky。

3. Findings 验证

  1. 证实:MessageId 判别值破坏。
    PR 在既有 health_ids 分段内插入 41 个 ID,后续固定值从 750/751/752 改为 791/792/793,整体偏移 41;测试断言被同步修改,未保持兼容值。

  2. 证实:信号处理器执行非 async-signal-safe 操作。
    runtime/terminal.rs:87 在信号处理器调用 remove_shell();该函数涉及 OnceLock、Mutex::try_lock、路径析构和 fs::remove_file。

  3. 证实:PID 复用会导致身份误判。
    run_registry.rs:293 仅用 kill(pid, 0) 判断存活;env_collectors.rs:800 据此认定 core 身份并在 owner 不存活时生成 kill {pid} 建议,未校验进程启动时间或可执行文件。

  4. 证实:production 文件超过布局阻断线。
    env_collectors.rs 本 PR 增加 493 行、删除 1 行,最终为 1190 行,超过 AGENTS.md 规定的 1000 行限制。

  5. 证实:底层模块反向修改 diagnostics 状态。
    shell_host/osc/routing.rs:32,72 和 adapter/cosh_core_service.rs:405 直接调用 run registry;更新路径持锁并执行文件写入/rename。

  6. 证实:24 小时日志计数仅为尾部样本。
    每个日志只读取末尾 256 KiB,却将结果公开为 logs.warns_24h、logs.errors_24h,高流量日志会静默漏计。

  7. 证实:crash 截断存在 UTF-8 切片 panic。
    shell 与 core 两处均以任意字节偏移执行 existing[keep_start..];偏移落入多字节字符时会在 panic hook 中再次 panic。

4. 补充测试

无。

5. 证据索引

未生成截图;证据目录为空。

6. 未运行 scope

  • workspace 全量测试
  • fmt、clippy
  • check-layout.sh
  • PTY/UI 手工验证
  • 文档链接检查

7. 结论

No-Go — focused cosh-core scope 两次稳定失败,且全部静态 findings 均被代码证据证实;静态评审结论无需修正。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:cosh-ng src/cosh-ng scope:documentation ./docs/|./*.md|./NOTICE

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants