Skip to content

[TestAgent][Nightly][ubuntu-agentsight] bug: interruption store 迁移顺序错误,旧库升级后 /api/interruptions 全部 503 #3314

Closed
#3378
Closed
[TestAgent][Nightly][ubuntu-agentsight] bug: interruption store 迁移顺序错误,旧库升级后 /api/interruptions 全部 503#3314
#3378
Assignees
chengshuyiviron-xiaojfeng18
Labels
component:sightsrc/agentsight/priority:p2Normal: planned product or engineering worktype:bugA confirmed defect or incorrect behavior

Description

@zhangtaibo
zhangtaibo
opened on Sep 17, 2026

Component

agentsight（Ubuntu 24.04 非 RPM 安装通道，anolisa install）

Bug Description

agentsight 0.12.x 的 interruption store 初始化失败，导致所有 /api/interruptions/* 端点返回 HTTP 503 {"error":"Interruption store not initialized"}，agentsight interruption 子命令报 no such column: conversation_id。

根因是 InterruptionStore::init_tables() 里针对旧库的 conversation_id 列迁移（ALTER TABLE ADD COLUMN）被放在了 CREATE INDEX IF NOT EXISTS idx_interruption_conversation ON interruption_events(conversation_id) 之后、且同批 execute_batch 被 ? 提前中断。任何由旧版 agentsight（尚无 conversation_id 列，即 <0.12.x 的库）创建的 interruption_events.db，升级到 0.12.x 后打开即失败，迁移代码永不到达（死代码）。

该问题由 AgenticOS Nightly 自动化测试在 2026-09-17 nightly（nightly-20260917-112309，anolisa main a9a3719）发现并定位。同一根因一次炸出整个 interruption API 用例族（list/count/stats/session-counts/conversation-counts/by-session/by-conversation/resolve/detail/event-history 等端点的"返回 200 列表"断言全部 503），以及 CLI interruption 子命令与 webui 依赖这些端点的页面。

Steps to Reproduce

可复现产品侧手工步骤（不依赖测试仓）：

# 1. 构造一个旧 schema 的 interruption_events.db（缺 conversation_id 列，
#    等价于任何 agentsight <0.12.x 版本创建的旧库）：
python3 - <<'PY'
import sqlite3
c = sqlite3.connect("/tmp/old_interruption_events.db")
c.execute(
    "CREATE TABLE interruption_events ("
    "id INTEGER PRIMARY KEY AUTOINCREMENT,"
    "interruption_id TEXT NOT NULL UNIQUE,"
    "session_id TEXT, trace_id TEXT, call_id TEXT, pid INTEGER,"
    "agent_name TEXT, interruption_type TEXT NOT NULL, severity TEXT NOT NULL,"
    "occurred_at_ns INTEGER NOT NULL, detail TEXT,"
    "resolved INTEGER NOT NULL DEFAULT 0,"
    "created_at DATETIME DEFAULT CURRENT_TIMESTAMP)"
)
c.commit()
PY

# 2. 用 agentsight CLI（带 --db 指向旧库）触发 interruption store 初始化：
agentsight interruption --db /tmp/old_interruption_events.db list --last 5
# → Error opening interruption database "/tmp/old_interruption_events.db":
#   no such column: conversation_id in
#   CREATE INDEX IF NOT EXISTS idx_interruption_conversation ON interruption_events(conversation_id);

# 3. 或直接访问 HTTP API（serve 进程在旧库上初始化失败后，store 为 None）：
curl http://127.0.0.1:7396/api/interruptions
# → {"error":"Interruption store not initialized"}  (HTTP 503)

Actual Behavior

旧库 + agentsight 0.12.1 下：

  • curl http://127.0.0.1:7396/api/interruptions → HTTP 503，body {"error":"Interruption store not initialized"}
  • /api/interruptions/count、/api/interruptions/stats、/api/interruptions/session-counts 等同样 HTTP 503
  • agentsight interruption list --last 5 → Error opening interruption database "...": no such column: conversation_id in CREATE INDEX IF NOT EXISTS idx_interruption_conversation ON interruption_events(conversation_id);
  • journalctl 反复出现 agentsight::server WARN Failed to open interruption store: no such column: conversation_id ... 与 agentsight::unified WARN Failed to initialize interruption store: ...（trace 侧与 serve 侧同时失败）

实现录路径稳定 2/2（每次调用都 503，非偶现）。

Expected Behavior

interruption_events.db 是持久化数据文件（文档承诺 30 天 / 100MB 保留，见 docs/user-guide/en/agent-observability/agentsight/data-and-storage.md），升级不应破坏既有库。旧库（缺 conversation_id）应在打开时被迁移补齐列，随后 interruption store 正常初始化，GET /api/interruptions 返回 HTTP 200 + 中断事件列表（见 src/agentsight/src/server/handlers.rs:3407 list_interruptions，store 就绪时应返回 200）。

Root Cause

src/agentsight/src/storage/sqlite/interruption.rs:66-100 的 init_tables()：

conn.execute_batch(
    "CREATE TABLE IF NOT EXISTS interruption_events ( ... conversation_id TEXT ... );
     ...
     CREATE INDEX IF NOT EXISTS idx_interruption_conversation ON interruption_events(conversation_id);
     CREATE TABLE IF NOT EXISTS process_exits ( ... );",
)?;   // <-- 旧库上 conversation_id 不存在，这里 CREATE INDEX 失败，? 立即返回 Err
// Migration: add conversation_id column for existing databases
let _ =
    conn.execute_batch("ALTER TABLE interruption_events ADD COLUMN conversation_id TEXT;");
Ok(())
  • 对已存在的旧库，CREATE TABLE IF NOT EXISTS 是 no-op；紧接着的 CREATE INDEX ... ON interruption_events(conversation_id) 因列不存在抛 no such column: conversation_id，?（第 96 行）把错误向上传播，init_tables() 直接返回 Err。
  • 而本可补齐该列的迁移 ALTER TABLE ... ADD COLUMN conversation_id（第 98-99 行）位于 ? 之后，永远执行不到，是死代码。
  • 返回值 new_with_path()（interruption.rs:62 store.init_tables()?）失败后，server/mod.rs:720-732 与 unified.rs:500-517 把 interruption_store 置为 None；handlers.rs:3412-3414 在 store 为 None 时返回 HttpResponse::ServiceUnavailable()（HTTP 503）并携带 {"error":"Interruption store not initialized"}。

引入 commit：c24f647d（"fix(sight): add interruption CLI and align conversation_id naming"，2026-04-24，liyuqing）。该 commit 在 CREATE TABLE 中新增 conversation_id 列并加了 idx_interruption_conversation 索引，同时补了迁移——但把迁移放在了会先失败的 ? 之后，属 regression（旧库升级路径被破坏）。当前 main（a9a37197）仍是该坏序，未修复。

触发条件：interruption_events.db 已经存在且缺 conversation_id 列（任何 agentsight <0.12.x 版创建的库，或非 RPM 通道预置镜像里的旧库）。全新库（CREATE TABLE 直接建出含 conversation_id 的表）不受影响，这也是该 bug 只在"升级/预置旧库"场景暴露、全新安装不暴露的原因。

Suggested Fix

把迁移 ALTER TABLE ... ADD COLUMN conversation_id 提前到依赖该列的 CREATE INDEX 之前（并拆分 execute_batch，避免同批 ? 提前中断），使旧库先补列再建索引：

     fn init_tables(&self) -> Result<(), Box<dyn std::error::Error>> {
         let conn = self.conn.lock().unwrap_or_else(|e| e.into_inner());
+        // 1) 建基础表（不含 conversation_id 索引）
         conn.execute_batch(
             "CREATE TABLE IF NOT EXISTS interruption_events (
                 ... conversation_id TEXT ...
-            );
+            );
+            CREATE TABLE IF NOT EXISTS process_exits (...);",
-            CREATE INDEX IF NOT EXISTS idx_interruption_session  ON interruption_events(session_id);
-            ...
-            CREATE INDEX IF NOT EXISTS idx_interruption_conversation ON interruption_events(conversation_id);
-            CREATE TABLE IF NOT EXISTS process_exits (...);",
         )?;
-        // Migration: add conversation_id column for existing databases
+        // 2) 迁移先行：旧库补 conversation_id 列（新鲜库重复列报错，已存在则忽略）
         let _ =
             conn.execute_batch("ALTER TABLE interruption_events ADD COLUMN conversation_id TEXT;");
+        // 3) 再建索引（此时 conversation_id 必存在）
+        conn.execute_batch(
+            "CREATE INDEX IF NOT EXISTS idx_interruption_session  ON interruption_events(session_id);
+             CREATE INDEX IF NOT EXISTS idx_interruption_occurred ON interruption_events(occurred_at_ns);
+             CREATE INDEX IF NOT EXISTS idx_interruption_type     ON interruption_events(interruption_type);
+             CREATE INDEX IF NOT EXISTS idx_interruption_agent    ON interruption_events(agent_name);
+             CREATE INDEX IF NOT EXISTS idx_interruption_resolved ON interruption_events(resolved);
+             CREATE INDEX IF NOT EXISTS idx_interruption_conversation ON interruption_events(conversation_id);",
+        )?;
         Ok(())
     }

已本地/ECS 验证根因（见 Actual Behavior）：对缺 conversation_id 的旧库直连 SQL 执行 CREATE INDEX ... idx_interruption_conversation ... 返回 no such column: conversation_id，而先 ALTER TABLE ADD COLUMN conversation_id 再建索引即可成功。

Environment

  • Ubuntu 24.04.4 LTS（非 RPM 通道，anolisa install 安装）
  • anolisa main：a9a37197
  • agentsight：0.12.1（/usr/local/bin/agentsight）
  • 受影响数据文件：/var/log/sysak/.agentsight/interruption_events.db（旧 schema，缺 conversation_id 列）

Additional Context

  • 影响面（同一根因，非逐个独立 bug）：/api/interruptions list/count/stats/session-counts/conversation-counts/by-session/by-conversation/resolve/detail/event-history 全部 503；agentsight interruption 子命令（list/count/stats）报 no such column: conversation_id；webui 依赖上述端点的页面（savings / skill-metrics / agent-health / genai 可视化等）在浏览器侧收到 503。
  • 稳定性：连续多次稳定复现（每次调用 503，非 flaky）。
  • 无 workaround 前提说明：全新库（首次安装、无历史数据）不受影响；验收修复可对旧 schema 库执行 agentsight interruption list 或 curl /api/interruptions 应返回空列表而非 503。

Activity

  1. zhangtaibo
    added
    priority:p2Normal: planned product or engineering work
    component:sightsrc/agentsight/
    type:bugA confirmed defect or incorrect behavior
    on Sep 17, 2026
  2. kongche-jbw
    assigned
    chengshuyi
    ,
    jfeng18
    and
    viron-xiao
    on Sep 17, 2026

  3. github-actions commented on Sep 17, 2026

    @github-actions
    github-actionsBot
    on Sep 17, 2026 – with GitHub Actions

    🔀 Issue Router

    问题涉及 AgentSight 的 interruption store 数据库迁移与中断 API。

    Evidence: 正文明确指向 AgentSight interruption 模块及其旧库迁移逻辑，属于 AgentSight 内部实现问题。

    This routing was applied by the ANOLISA Issue Router through GitHub Actions.

  4. zhangtaibo commented on Sep 17, 2026

    @zhangtaibo
    zhangtaibo
    on Sep 17, 2026
    CollaboratorAuthor

    补充证据(2026-09-17 nightly run 复确认,连续稳定):

    • journalctl smoking gun:agents 服务启动期反复 WARN Failed to initialize interruption store: no such column: conversation_id(CREATE INDEX idx_interruption_conversation ON interruption_events(conversation_id) 在旧 schema 库上执行失败)→ store 未初始化 → /api/interruptions/* 全量 503。
    • 影响面:该 run Ubuntu 通道 agentsight 套件 ~50 例连锁失败(API 断言 503==200 ×23、CLI "Error opening interruption database" ×10、响应结构 KeyError ×7、WebUI 页面 5xx 断言 ×10)。
    • 触发条件确认:-opt 定制镜像预装旧版 agentsight(遗留旧 schema interruption_events.db),nightly 从源码构建安装新版后首启即触发——即"旧库升级"路径,与 issue 标题的迁移顺序根因一致。RPM 通道(ALinux4,全新库)同版本 0 失败,佐证仅升级路径受影响。
    • 测试侧动作:受影响用例将加条件 xfail(strict,reason 引本 issue),修复后 XPASS 提示移除。

  5. zhangtaibo commented on Sep 17, 2026

    @zhangtaibo
    zhangtaibo
    on Sep 17, 2026
    CollaboratorAuthor

    补充同根因的下游影响面(nightly-20260917-164503 复现确认):

    启动 OOM 恢复路径也被同一迁移顺序 bug 打断。unified.rs:521-522 的 recover_oom_events(istore, ...) 仅在 if let Some(ref istore) = interruption_store 时执行;store 初始化失败后 interruption_store = None,导致恢复扫描整段被跳过,注入 dmesg 的 OOM 事件一个都不会写回 DB。

    实测(ECS 47.83.182.36,agentsight 0.12.1):

    • interruption_events.db 现为旧 schema(无 conversation_id 列)
    • 注入 Out of memory: Killed process <synthetic-pid> (oom-recovery-contract) ... 到 /dev/kmsg → systemctl restart agentsight → count = 0
    • journalctl 无任何 "OOM recovery:" 日志(未被调用),反复 "Failed to initialize interruption store: no such column: conversation_id"

    因此 test_oom_recovery_memcg_format.py 的 summary-line(positive control)与 memcg structured 两个 case 都报 "recovered 0 events"。注意:memcg case 的断言消息仍硬编码引用 #3130,但 #3130 是"parse_dmesg_lines 不识别 memcg 结构化格式"的缺口,已由 PR #3135(commit 28f5f0b)修复;当前失败的真实根因是本 issue 的迁移顺序 bug,与 #3130 无关。修复本 issue 后这两个 OOM recovery case 应一并恢复。

  6. zhangtaibo commented on Sep 17, 2026

    @zhangtaibo
    zhangtaibo
    on Sep 17, 2026
    CollaboratorAuthor

    Additional manifestation of this same root cause (2026-09-18 nightly nightly-20260918-020051): the OOM startup-recovery path is also silently disabled, not just the /api/interruptions 503.

    Because interruption_store is None, recover_oom_events is never invoked — src/agentsight/src/unified.rs:521-523 gates it behind if let Some(ref istore) = interruption_store. Net effect: both cases in test_oom_recovery_memcg_format.py fail with COUNT=0:

    Independent verification on ECS 8.217.208.136 (agentsight 0.12.1, Ubuntu 24.04): manually running the dead migration ALTER TABLE interruption_events ADD COLUMN conversation_id TEXT then restarting the service and injecting a summary OOM line yields COUNT=1 (recovered). Confirms the fix in ### Suggested Fix also restores OOM startup recovery.

  7. zhangtaibo commented on Sep 19, 2026

    @zhangtaibo
    zhangtaibo
    on Sep 19, 2026
    CollaboratorAuthor

    补充下游影响(同根因, 追加记录, 勿重开):同一 migration-order bug 也击穿了 OOM startup-recovery 路径, 不只是 /api/interruptions 503。

    修复 migration 顺序后 recover_oom_events 会一并恢复, 无需单独处理。

  8. chengshuyi
    closed this as completedin #3378on Sep 20, 2026
  9. haosanzi
    added 2 commits that reference this issue on Sep 20, 2026

    fix(sight): add conversation_id column before its index (agentic-os-o…

    2c276dd

    fix(sight): add conversation_id column before its index (agentic-os-o…

    4528134

  10. zhangtaibo commented on Sep 20, 2026

    @zhangtaibo
    zhangtaibo
    on Sep 20, 2026
    CollaboratorAuthor

    Additional downstream impact + recurrence note (Nightly 2026-09-21, ubuntu-agentsight 0.12.1):

    The OOM startup-recovery cases are also blocked by this exact root cause:

    • tests/test_oom_recovery_memcg_format.py::TestOomStartupRecoveryDmesgFormats::test_startup_recovery_parses_summary_killed_process_line
    • tests/test_oom_recovery_memcg_format.py::TestOomStartupRecoveryDmesgFormats::test_startup_recovery_parses_memcg_structured_line

    Both assert that a service restart rewrites an agent_crash event for an OOM-killed pid (via recover_oom_events). But recover_oom_events is gated on a successfully-initialized interruption store — unified.rs:501-523 only calls it if let Some(ref istore). With 0.12.1 the store init fails, store is None, and recover_oom_events never runs → 0 recovered events (both cases fail with recovered 0 events, not a parsing error — the summary/structured dmesg parsing itself already works).

    The fix PR #3378 (merged 2026-09-20) covers this, but is not yet in any released sight version (git tag --contains 2c276dd6a returns no sight/* tag). The Ubuntu channel still installs sight 0.12.1, so these two cases keep failing (consecutive_days=5 as of 2026-09-21). No new issue/PR filed — this is the same root cause awaiting the next sight release.

  11. jfeng18
    added a commit that references this issue on Oct 5, 2026

    fix(sight): migrate conversation_id before its index

    6541a18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

component:sightsrc/agentsight/priority:p2Normal: planned product or engineering worktype:bugA confirmed defect or incorrect behavior

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions