Skip to content

sqlite3.ProgrammingError: Cannot operate on a closed database in hourly memory maintenance (292 tracebacks/day), also hits conversation-time capture_turn #484

Description

@lccstc

Summary / 摘要

The hourly memory maintenance task repeatedly operates on closed SQLite connections, producing hundreds of tracebacks per day. It also occasionally hits the conversation-time capture path, which silently drops raw memory events.

每小时一次的记忆维护任务在已关闭的 SQLite 连接上执行,单日产生数百次 sqlite3.ProgrammingError;偶发打在对话时的 capture_turn 路径上,导致原始记忆事件静默丢失。

Environment / 环境

  • octop 0.9.28 (one-line installer, venv under /opt/octop/venv)
  • harness-memory 0.9.7 / harness-agent 0.9.27
  • Python 3.12.14 · Linux x86_64 (TencentOS Server 4, kernel 6.6) · 2 vCPU / 2GB RAM
  • Default SQLite backend, single octop run process

Observed behavior / 现象

On 2026-08-30 the daily log contained 292 sqlite3.ProgrammingError tracebacks. Stack clusters into two groups:

(1) Hourly maintenance pass (runs ~:21 each hour, logged as memory.maintenance done gc_err=ProgrammingError... vacuum_err=ProgrammingError... gc_rows=None pages=None):

harness_agent/middleware/memory.py:713  _run_reclaim_pass
harness_memory/core.py:394              list_candidates
harness_memory/pipeline/lifecycle/gc.py:177        _gc_rejected_candidates
harness_memory/pipeline/lifecycle/vacuum.py:215    nudge_vacuum
harness_memory/pipeline/lifecycle/vacuum.py:298    _sqlite_nudge_vacuum
harness_memory/storage/backends/sqlite.py:1165     list_candidates
harness_memory/storage/backends/sqlite.py:125      _conn
sqlite3.ProgrammingError: Cannot operate on a closed database.

(2) During real conversations (6 hits on the same day) — this one can silently lose memory writes for the turn in progress:

harness_memory/service.py:171  capture_turn
harness_memory/core.py:323     add_raw_batch
harness_memory/storage/backends/sqlite.py:949  save_raw_batch
sqlite3.ProgrammingError: Cannot operate on a closed database.

Notes / 备注

  • Without any process restart, later runs on the same day became clean again (memory.maintenance done gc_err=None vacuum_err=None), which suggests a race between connection pooling/closing and the background maintenance task rather than a permanent failure — possibly the pool closes a connection while _run_reclaim_pass still holds a reference, or the task grabs a connection right after idle-close.
  • Suggestion: acquire a fresh connection per maintenance call (or liveness-check + single retry), and log a visible warning when capture_turn fails, since those failures are currently silent data loss.

同一个进程未重启的情况下,当天晚些时候的维护任务又恢复正常(gc_err=None),更像连接生命周期竞态而非持久性故障。建议维护任务每次从连接池重新取连接(或做存活检查+一次重试),并对 capture_turn 失败输出显式告警——目前是静默丢数据。

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions