feat(memory): scrub high-entropy secrets before persisting extracted memories + indexing VectorDB

[lg320531124]

What happened

While debugging an OV (OpenViking memory server) configuration, an API key was pasted into a Claude Code conversation that OV's auto-capture hook records. OV captured the full message stream into a session (raw messages.jsonl, expected). The memory extraction pipeline then copied the key verbatim into a durable structured memory under data/viking/.../memories/events/<date>/<name>.md and indexed it into the vector store.

Result: the secret became retrievable across future sessions via find / search — a vector recall for an unrelated token surfaced the memory and returned the plaintext key into the LLM context.

Expected

The extraction pipeline's quality gates (the read-before-write + 6-gate filter that decides what becomes a durable memory) should prevent secrets from surviving into curated memories and the vector index.

Raw session capture containing a secret is expected — that's raw conversation, and redacting there would alter the raw-capture contract. But a curated, durable, vector-indexed memory containing a secret contradicts the gates' purpose: the whole point of the filter is to discard what should not be persisted.

Repro

1. Run a session where one message contains a test-shaped secret, e.g. sk-test-FAKE0123456789abcdef.
2. Trigger extraction (session commit / extract / SessionEnd).
3. Inspect data/viking/.../memories/events/<date>/ — an extracted memory contains the secret verbatim.
4. search / find for a token that appears near the secret — the vector store returns that memory, surfacing the secret into context.

Proposal

Add a secret-scrub gate between LLM extraction and persistence:

• Regex patterns for common secret shapes:
  • sk-[A-Za-z0-9]{16,} (OpenAI-style)
  • AQ[A-Za-z0-9_-]{20,} (Gemini-style)
  • xox[baprs]-[A-Za-z0-9-]+ (Slack)
  • Bearer\s+[A-Za-z0-9._-]+
  • high-entropy hex / base64 of length ≥ 32
• On match in an extracted memory body: replace with REDACTED_SECRET (or skip vector-indexing that memory, or flag it for review).
• Configurable via env / ov.conf: pattern list + on/off, so users can extend for internal key shapes; default conservative to avoid false-positive scrubbing of legitimate tokens / UUIDs / commit SHAs.

Scope

Extraction layer only. Session-layer raw capture (messages.jsonl) intentionally unchanged — redacting there would break the raw-capture contract and make raw logs useless for debugging.

Caveat

Secret detection has false-positive risk (commit SHAs, content hashes, long UUIDs can look high-entropy). Suggest opt-in initially + an allowlist for known-safe patterns, so legitimate identifiers are not mangled.

Why I'm filing

I have two open PRs in the auto-capture / compaction direction (#2874, #2853). This issue is the privacy complement to that storage work — reducing what gets captured/stored is half the story; ensuring secrets don't survive curation into vector-retrievable memory is the other half. Happy to contribute a PR if there's appetite for the approach above.

[xg-gh-25]

This is exactly the right layer for secret filtering. The extraction-to-curation boundary is where quality gates belong — after raw capture (preserving session integrity) but before persistence into searchable, long-lived memory.

We hit this same problem in SwarmAI's memory pipeline when debugging showed API keys surviving into cultivated Knowledge/ entries that became retrievable across sessions. Our solution mirrors your proposal:

1. Gate placement: Between LLM extraction and write-to-disk, not in the raw session capture. The raw messages.jsonl contract is sacred — redacting there breaks debugging. But a curated memory with a secret is a quality-gate failure.

2. Pattern library: We use regex anchors for common shapes (OpenAI sk-, Anthropic sk-ant-, AWS AKIA, Slack xox*, GitHub PAT ghp_) + entropy threshold for catch-all. The allowlist idea for commit SHAs / UUIDs is critical — we maintain a KNOWN_SAFE_PATTERNS set (40-char hex with git commit context, RFC 4122 UUIDs in schema fields) to prevent false-positive mangling.

3. Configurable scrub action: We offer three modes via MEMORY_SECRET_POLICY:

• redact: replace with [REDACTED_SECRET] (default, preserves memory structure)
• skip: drop the entire memory from vector index (conservative, may lose context)
• flag: write memory but add _contains_secret: true metadata (review queue)

4. User extension: ENV-driven pattern list (MEMORY_SECRET_PATTERNS) for internal key shapes. Example: ^CORP_[A-Z0-9]{32}$ for company-specific tokens.

The false-positive risk is real. Our data: ~2% of legitimate content hashes / JWT segments / base64-encoded non-secrets trigger entropy thresholds. The allowlist + opt-in default keeps this manageable.

One addition to your proposal: audit logging. When a secret is scrubbed, we log {timestamp, session_id, memory_candidate_hash, pattern_matched, action_taken} to a separate audit file (never indexed). This lets us tune patterns and catch false negatives without exposing the actual secret.

If you're building the PR, happy to share our regex library and test cases (synthetic secrets + edge cases like commit SHAs that look like keys). The entropy threshold calibration is the hardest part — we settled on ≥4.5 bits/char over ≥32 chars after testing against 10K real memories.

---

Context: we run an autonomous agent with persistent memory cultivation and hit this exact extraction-layer leak. SwarmAI. Discussion: T-MEM Memory Sovereignty

[lg320531124]

Thanks for the detailed reply — this is exactly the kind of consideration that should come after actually hitting and attributing this leak in OpenViking. A few of your points sharpened what my initial proposal left too thin:

• Audit log: the real addition. My original only considered redact; logging {timestamp, session_id, pattern_matched, action_taken} to a separate non-indexed audit file is what makes it possible to tune patterns later without re-exposing the secret.
• Three scrub actions (redact / skip / flag): better than a single replace. flag → review-queue is a good middle ground — keeps the memory available for investigation rather than silently dropping it.
• Entropy threshold calibration: your measured ≥4.5 bits/char over ≥32 chars against ~10K memories is exactly the data I don't have yet — it saves me from shipping a catch-net that would mangle legitimate base64/JWT fragments.
• Allowlist: I gestured at this in the caveat, but your KNOWN_SAFE_PATTERNS framing is clearer; the commit-SHA-looks-like-a-key false positive is real.

I'd gladly take you up on sharing the regex library + test cases — synthetic secrets + edge cases (especially commit SHAs that look like keys) would seed the PR I'm putting together. If you can paste them here or point at a file in SwarmAI, I'll credit the source.

Plan I'll bring in: MEMORY_SECRET_POLICY (redact | skip | flag), audit log separate from the vector index, ENV-driven MEMORY_SECRET_PATTERNS + KNOWN_SAFE_PATTERNS, opt-in by default. If the OpenViking maintainers are amenable to the direction, I'll link the PR to this issue.

[lg320531124]

Empirical reproduction on v0.4.7 (latest release).

Filing this as a weight-adding data point, not a new ask — the original proposal is still the right scope.

Repro path. Upgraded to v0.4.7 (released 2026-07-02). Within ~3 h of normal use, no special trigger: a sub-agent read ov.conf to pick up root_api_key for a curl smoke test, the session was committed, and the v0.4.7 extractor copied the key verbatim into memory summaries/bodies and session archives.

Leak surface (27 files).

• 12 long/short-term memory .md — including .overview.md (the directory index that lists memory titles), trajectories, experiences, and entities. Several of these carry the key in the body; .overview.md exposes it in a listing context.
• ~15 session archives — messages.jsonl + memory_diff.json per archived session.

Root cause (from reading the source). v0.4.7 ships the self-evolution / experience-memory refactor (commit fd73dcf, 2026-07-03, "自进化（经验记忆）框架重构"). The new extraction flow regenerates memory summary/body from LLM context but performs no high-entropy string detection or redaction before write-to-disk + VectorDB upsert. The 6-gate quality filter judges memory salience, not secret content — so a key that the LLM deems "salient" survives curation by design.

Impact, measured.

• VectorDB recall: a semantic search for the unrelated token "openviking 升级" returns a memory containing the key at score ~0.7 — the key is re-injected into LLM context on recall.
• Index exposure: the key appears in .overview.md, so any directory-listing / overview-read path surfaces it without needing a vector query.

Suggested fix points (leaving design to maintainers).

1. Add a high-entropy scrub step in the extractor before writing memory .md and upserting VectorDB — regex anchored on ov-root- prefix plus a generic high-entropy token detector (the sk-/AKIA/xox*/ghp_ library already discussed above; an OV-internal prefix pattern is the gap here).
2. Backfill / migration: scrub already-indexed VectorDB entries for affected deployments (the key is already in the index for existing users — a forward-only fix leaves prior leaks retrievable).
3. Session-archive path: decide whether messages.jsonl / memory_diff.json should be scrubbed at archive time, or remain raw per the original raw-capture contract. At minimum, memory_diff.json (curated diff, not raw capture) looks like it should be in scope.

No status/label changes requested. Happy to contribute a PR for (1) + a migration script for (2) if maintainers confirm the direction.