Skip to content

Conversation judge (in progress) - #508

Merged
mchestr merged 31 commits into
mainfrom
claude/sleepy-feynman-bo0yp4
Nov 2, 2026
Merged

mchestr merged 31 commits into
mainfrom
claude/sleepy-feynman-bo0yp4

Conversation

@mchestr

@mchestr mchestr commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Summary

This is the conversation judge from docs/superpowers/plans/2026-11-01-conversation-judge.md, built task by task: a subagent implementer per task, an independent task review, a security review on the sensitive tasks, and up to five fix rounds.

Not ready to merge yet. Tasks 15–17, the final whole-branch review and /security-review are still running.

Merging at 421b672 would ship the tool log, the review job, the actions, the nightly sweep (off by default), the loaders, the MCP read tool, the conversation page's review section and the Findings tab. It would not yet:

  • clear a conversation's findings and review summaries when the conversation ends (Task 16 — the privacy half of the design);
  • end an unlinked Discord member's leftover conversation (Task 16);
  • include the judge settings UI (Task 15);
  • include the evals and E2E (Task 17).

It builds on #510, which is merged. main is merged in at 421b672, which changes no file, so this diff now shows only the judge.

Done so far

  1. Task 2: schema and migration (20270121110000_add_conversation_judge). It adds:
    • ConversationReview and ConversationFinding
    • ChatConversationMessage.toolLog and ChatConversation.reviewLockedAt
    • LLMUsage.conversationReviewId (SetNull) and the Config judge fields
    • the encrypted and free-text registrations, the judge purpose, the judge-spend lease name, factories, and changelog.d/conversation-judge.md
  2. Task 3: redacted tool log. Each admin assistant turn stores its tool calls encrypted, redacted, then capped. The log never reaches the browser, the MCP chat tool or Discord.
  3. Task 4: the judge output schema and fingerprint.
  4. Task 5: the fenced, capped judge prompt builder.
  5. Task 6: evidence verification, against the text the judge was actually sent.
  6. Task 7: findings persisted by fingerprint, never deleted. A repeat never downgrades verified evidence.
  7. Task 8: the judge's own daily spend cap and the judge-spend Redis lease.
  8. Task 9: the review job (judge:review). It covers:
    • run-token fencing
    • re-delay on a taken lease without spending an attempt
    • retries for PROVIDER and ABORTED
    • reaping stuck reviews
  9. Task 10: reviewConversation, getConversationReviewStatus and setFindingStatus, with the shared enqueueReview.
  10. Task 11: the nightly sweep (03:17 server time, off by default, at most 50 reviews), its schedule and saveJudgeSettings.
  11. Task 12: the finding loaders and the list_conversation_findings MCP read tool.
    • The loaders are lib/data/conversation-findings.ts and lib/data/conversation-review.ts. Reads of a member's conversation are audited as conversation-review and conversation-findings.
    • The tool clips its text so a default call fits the MCP result limit.
  12. Task 13: the conversation page's review section. It has the review button and status, the finding cards with the message each points at, and triage.
  13. Task 14: the Findings tab on AI activity, with filters, groups, stat tiles and the Calls review badges.

Still to come

  • Task 15: the judge provider tab and the Conversation review settings
  • Task 16: privacy, retention, disclosure, metrics and docs
  • Task 17: eval fixtures, E2E and the full gate
  • the final review, its single fix pass and /security-review

The final fix pass already includes:

  • verify quotes from earlier messages against the text as sent
  • cap merged claims
  • a prompt rule that text cut at [truncated] is unknown
  • count a repeat that keeps the stored evidence as seen
  • the sweep's skip while nightly reviews are off
  • the deferred naming and baseline items

Checklist

  • Matches the components sheet (/dev/ui); visual diff attached (the UI from Tasks 13–15 changes the AI activity screenshots; the Visual Baselines refresh is still needed)
  • npm run design:check passes
  • New interactive elements have a data-testid
  • No golden snapshot, .claude/skills/** or .mcp.json changes

🤖 Generated with Claude Code

https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

@claude

claude Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Claude finished @mchestr's task in 34s —— View job


Review of the conversation judge PR (draft, in progress)

I read the review job, persist.ts, verify.ts, and the conversation-review and judge-settings actions. I didn't run the test suite or the other ~90 changed files, so this is a focused read, not a full audit. I found nothing blocking. Most points below are already on your deferred list.

What looks good

  • Run-token fencing: startRun increments runToken in a transaction, and every later write is conditional on { id, status: 'running', runToken }. A stalled or redelivered job can't overwrite a newer run.
  • Persistence: persistReview takes the conversation row lock before writing, so a review finishing and a conversation ending serialise. Throwing ReviewSuperseded rolls the transaction back cleanly.
  • Failure handling: judgeFailed retries only PROVIDER and ABORTED. The lease-busy path re-delays without spending an attempt.
  • Logging and audit: logs and audit events carry ids, counts and codes, never finding or summary text. The saveJudgeSettings and setFindingStatus actions follow the adminAction conventions.
  • Evidence verification: it matches against sentToolText, the text as the judge actually received it.

Possible issues, ordered by likely impact

  1. Unhandled throw before the lease. In conversation-review.ts:130-131, getAdminAssistantContext and buildJudgePrompt run outside any try/catch.

    • If either throws (the owner's role changed, a tool-catalog error), the job fails but the review stays running until the reaper marks it stale. The admin sees a spinner for the whole stale window. If the reaper's window is long, wrap these in a try/catch that calls markReviewFailed(reviewId, 'JOB_FAILED', run.runToken). The same applies to generateStructured throwing instead of returning { ok: false }.
    • Please confirm what stale.ts does here and add a test for that path.
  2. saveJudgeSettings can turn the sweep on with no judge model.

    • The judgeNightlyEnabled save succeeds even when no judge provider is configured. The sweep would then enqueue reviews that all fail with NOT_CONFIGURED. reviewConversation already checks this and returns UNAVAILABLE.
    • Consider refusing, or showing a warning, when the toggle is on and there is no provider. Skip this if the sweep already skips on no provider.
  3. Merged claims (verify.ts:99) join with " / ", uncapped and not de-duplicated. You've already scheduled both fixes in the final pass.

  4. earlier_message verification (verify.ts:83-84) matches the stored message.content, not the text as sent. This is the same known item, scheduled for the final pass. When you fix it, keep the ctx.window.has(message.index) filter.

  5. Triage after the conversation ends. setFindingStatus reads then updates by id only. After Task 16, NOT_FOUND comes from the zero-count path, as you described. Add a pg test for the ordering where the finding is deleted between the read and the update, since a findUnique hit followed by count === 0 is the case you're relying on.

  6. getConversationReviewStatus calls refreshView() on every terminal read. You've shown that the poller stops after the first terminal poll. A direct caller, such as another client of the action, would trigger a refresh on every call, so keeping the one-line note on the action is worthwhile.

  7. Test coverage. The unit tests for the job, persist, verify and actions look thorough from their names and sizes. I'd want pg coverage for these before leaving draft, and you've already planned most of it for Task 16:

    • the (conversationId, fingerprint) unique constraint under two concurrent reviews
    • an ending racing persistReview
    • judge spend surviving an ending (conversationReviewId SetNull)

Housekeeping

  • The PR checklist still has the visual-diff item unchecked. The Visual Baselines refresh should wait until the AI activity UI settles in the final pass.
  • The orphaned admin-llm-usage.png baselines and the deferred naming items (icons.llmUsage and similar) are still open, as noted in the earlier threads.
  • My working tree for this run had an uncommitted CLAUDE.md edit that removes the lib/judge/ line from the file map. It isn't in the PR diff, and I left it alone.

Overall the design is careful and the concurrency story holds up. Only point 1 seems worth a change before merge. Points 2 and 7 are small additions.
· Branch: claude/sleepy-feynman-bo0yp4

mchestr commented Nov 1, 2026

Copy link
Copy Markdown
Owner Author

Thanks. I checked each point against the code, and a34ba52 addresses them:

  1. Ending a conversation and judge spend: real. Ending a conversation now keeps its review rows but drops their text: it deletes the findings, clears the summaries and fails any running review with ENDED. That way LLMUsage.conversationReviewId (onDelete: SetNull, now stated in Task 2) keeps the link the cap and the judge filter count on. This is the same reason lib/chatbot/history.ts keeps an ended conversation's row. The retention delete removes them months later. Keeping the rows created a race of its own: a review could finish after the ending and write quotes back. So persistReview now locks the conversation row first, as saveTurn does, through a new ChatConversation.reviewedAt, and throws ReviewSuperseded if the conversation has ended. The spec's §3, §4 and §5 are updated, and there are pg tests for the spend and for both orderings.
  2. The sweep doesn't create review rows: real. A shared enqueueReview (lib/judge/enqueue.ts, Task 10) now checks, reaps stale reviews, dedupes and creates the row in one transaction, then adds the job after commit. Both the action and the sweep use it, and tests cover the skip and reap cases.
  3. spendDay: right, it's async and private (lib/llm/budget.ts:69). Task 8 now exports it and uses (await spendDay(now)).start.
  4. Review Focus 1: added the Task 6 test and fixed the cross-reference.
  5. Description numbering: fixed.
  6. Invite codes: confirmed. getAssistantInvites (lib/data/assistant-data.ts:38) returns only codeEnd: code.slice(-4). Task 3's test now names it.
  7. Failure classes: only LEASE_BUSY, PROVIDER and ABORTED retry, and the review goes back to queued during backoff. Everything else fails with its code as is (NOT_CONFIGURED replaces my made-up CONFIG). Attempts are 3 from DEFAULT_JOB_OPTIONS, and the final-attempt hook condition is stated.
  8. Already-reviewed messages: those findings aren't dropped. Context messages are in window on purpose, because the spec allows a finding there when a later message makes an earlier problem worse, and the fingerprint suppresses repeats. A test now pins that.
    9–10 and process: the SelectTrigger placement, the floors in the commit message, and the security-review targets are all added.

Generated by Claude Code

mchestr commented Nov 1, 2026

Copy link
Copy Markdown
Owner Author

Rewind E2E Tests failed on a78b754, and the failure isn't this PR's.

  • What failed: [rewind-e2e-webkit] e2e/rewind/story.spec.ts:66, "Rewind story › with autoplay off › the Table sheet opens on a chart page and closes". expect(sheet).toBeHidden() timed out at 5 s on all three tries. e2e/rewind/editions.spec.ts:16 was flaky (it passed on retry), and 232 tests passed.
  • Why it isn't this PR's: this PR changes only two Markdown files under docs/superpowers/, with no app code, styles or config. The latest Checks run on main (41345793105) is green.
  • Next: GitHub refused to re-run the failed jobs while the rest of run 41351270899 is still going, so I'll re-run them once when it finishes. If the WebKit story test fails again, it's a WebKit timing problem in the Rewind sheet's close, independent of this PR, and I'll report it rather than widen this docs PR.

Generated by Claude Code

@mchestr mchestr changed the title Implementation plan: conversation judge Conversation judge: plan, and rename LLM usage to AI activity Nov 1, 2026

mchestr commented Nov 1, 2026

Copy link
Copy Markdown
Owner Author

Thanks. On the optional points:

  1. icons.llmUsage: it's an internal icon-map key, so it stays for now. I've added it to the deferred naming items for the final review, along with the record page's "LLM request" heading and the three page component names, so they're settled together.
  2. Internal names: these are deliberate. Task 1's Step 3 in the plan allows the remaining internal names: lib/data/llm-usage.ts, lib/admin/analytics/llm-usage.ts, lib/mcp/read-tools-llm.ts, the LLMUsage model and the llm-usage-detail audit key.
  3. Axe baseline: checked. It has 40 entries before and after, and each of the three moved route keys (admin /admin/ai, …/conversation/a11y-conversation and …/visual-llm-usage-1) carries the same values as before.

The description was updated before this review finished. It now describes Task 1 and says the judge commits will land here until a branch decision is made.


Generated by Claude Code

…fter its filters

The sixth plan review: two concurrent enqueues could each create a review, so
enqueueReview locks the conversation row first (ChatConversation.reviewedAt is
renamed reviewLockedAt, a lock column nothing reads). The sweep's 50 limit now
applies after sampling and the failure skip, the purpose grep covers every
two-purpose assumption, and new tests pin the Error heuristic, a rejected
moveToDelayed, an ending's encrypted review columns and inert MCP text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

mchestr commented Nov 1, 2026

Copy link
Copy Markdown
Owner Author

Thanks. On the three points:

  1. Orphaned baselines: agreed. The two admin-llm-usage.png files (desktop and mobile) are no longer referenced by any spec. They're in the deferred list for the final review's fix pass, which will delete them unless the Visual Baselines refresh has already removed them.
  2. Naming: icons.llmUsage, the record page's "LLM request" heading and the sibling component names will be settled in that same pass, before this leaves draft.
  3. Splitting: I agree it's easier to review split. Separate branches for the rename and the judge are waiting on Mike's answer. Until then, this session can push only to this branch.

Generated by Claude Code

ConversationReview and ConversationFinding with their enums, the
assistant message tool log, the judge's Config fields, LLMUsage's link
to a review and the conversation's review lock column, in one additive
migration (20270121110000_add_conversation_judge). The judge's text
columns are encrypted free text, LLMPurpose gains "judge", LeaseName
gains "judge-spend", and the factories build the new rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
runConversationLoop now records each tool call's name, arguments, output
and an ok hint (isToolSuccess: an output not beginning "Error"), and
passes them through capToolLog before they leave the loop: credentials
are redacted with lib/mcp/redact.ts first, then args are cut to 2 KB,
each result to 8 KB and the turn to 32 KB, marking anything cut
truncated. runChatbotForUser returns { response, toolLog } so the log
never rides on ChatMessage or ChatResponse; the admin chat action hands
it to saveTurn, which stores it on the answer's encrypted toolLog
column, and the Discord path ignores it, so Discord's plaintext session
JSON never sees it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
…s an empty log

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

mchestr commented Nov 1, 2026

Copy link
Copy Markdown
Owner Author

Thanks, both reviews were useful. I checked each point against the code:

  • isToolSuccess vs isToolError: this is being fixed now. It was a ruling on the task review's plan-mandated finding: isToolSuccess(output) becomes !isToolError(output), with tests for ROLE_TOOL_NOT_PERMITTED, {"success": false} and the "Error 404" heuristic. The commit message was ahead of the code, which lands in Task 3's fix round.
  • ok after the scrub: this is covered once that change lands. DISCORD_TOOL_UNAVAILABLE and ROLE_TOOL_UNAVAILABLE are both "Error: that tool isn't available right now." (lib/chatbot/executors/index.ts:36, role-scrub.ts:35), and the *_NOT_PERMITTED refusals are JSON { error }. isToolError catches both forms, so every scrubbed failure logs ok: false.
  • arguments type: it's never an object. lib/llm/chat.ts:184 always sets arguments: JSON.stringify(call.input ?? {}).
  • truncated with empty args under an exhausted budget: this is harmless, so it's deferred to the final review's list.
  • parseToolLog's @internal tag: dropping it is in the review-job task's brief (Task 9).
  • The rest is known: the security review will cover capToolLog, the new ENCRYPTED_FIELDS entries and the Discord log drop. The pg coverage of the unique fingerprint and the cascades is planned for Tasks 7, 10 and 16. The deferred naming and baseline cleanup lands before this leaves draft, and the split waits on Mike.

Two more changes are in the same fix round: an empty tool log ([]) for a turn that called no tools, so a stored null means only "recorded before the tool log", and String(output ?? '') when building log entries, so a logging side path can't break an answer.


Generated by Claude Code

…he tool name

- isToolSuccess is !isToolError, so JSON refusals and error outputs log as ok: false
- runConversationLoop returns a toolLog for every answered turn; serializeToolLog
  stores '[]' for an empty log and null only when none was captured
- each entry's result is String(output ?? ''), so a non-string output cannot fail the turn
- capToolLog redacts the tool name and cuts it to 100 characters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
judgeOutputSchema caps a review at 20 findings and a 600-character summary,
with text caps (evidence 4000, claim and suggestion 2000, tool 100) well above
the 300-character quotes the prompt asks for. fingerprintOf hashes
type|tool|messageId only, so a reworded claim still matches. Both are knip
entries until the review job imports them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
OpenAI strict structured outputs need every property listed in required;
an optional tool left it out and would fail the judge call with a 400.
A finding no tool applies to now carries tool: null. A test pins that the
JSON Schema is all-required with additionalProperties: false.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
buildJudgePrompt fills a 120,000-character budget in order: the assistant's
prompt and catalog, the newest turn whole (its pieces cut to a fair share only
when it alone passes the cap), the "Already raised, don't repeat" block, then
earlier turns newest first with tool results cut to 2 KB, and a line counting
the turns left out. Each message keeps its absolute index, null and empty tool
logs are labelled, and messages up to the last done review are marked as
context. Every fence close tag inside the content becomes <\/tag (JSON's own
escape, so a tool result still parses), so each block closes once.

getAdminAssistantContext gives the judge the prompt and toolset the owner gets
today, a Support owner's smaller toolset and scope section included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
@mchestr
mchestr marked this pull request as ready for review November 2, 2026 00:09
verifyFindings checks each finding's quote against the text the judge was
sent: tool results leaf by leaf, then as sent text (cut to 2 KB for earlier
turns), earlier messages by lower index, catalog tools by name. A finding
outside the prompt's window is dropped; a failed check keeps it as
unverified; duplicates on one fingerprint merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
@mchestr mchestr changed the title Conversation judge: plan, and rename LLM usage to AI activity Conversation judge (in progress) Nov 2, 2026
@mchestr
mchestr marked this pull request as draft November 2, 2026 00:12
The judge call is a strict structured output, so every finding key is
required and a finding no tool applies to carries tool: null, as
lib/judge/schema.ts already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
Brings in #504, #507 and #509. The one conflict: main moved isToolError
into lib/chatbot/executors/tool-output.ts, so the executors keep only
isToolSuccess, which reuses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

mchestr commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Thanks. I checked each point against cd45006:

  1. catalog counting as verified: this stays as it is. The UI never labels a finding "verified" or "confirmed". The evidence check only decides what the default views count, and a failed check shows as "Unverified quote" (Tasks 12 and 13). A catalog claim can only be checked against the catalog: a wrong_tool must name a real tool, and a missing_capability must name none. The JSDoc table in verify.ts says so.
  2. earlier_message against cut text: this is real but narrow. JudgeMessage.content is the stored text. build-prompt.ts cuts message text only when the newest turn alone passes the 120,000-character cap, and earlier turns always go whole. In that case, a quote from the cut part of the newest turn's user message could still verify. The fix rides a later push on this branch, before it leaves draft: match against the text as sent, the way tool results already use sentToolText.
  3. Merged claim length: this is real. Three or more findings with the same fingerprint can join to more than 2,000 characters. The column is unbounded encrypted text, so nothing breaks, but the same push caps the merged claim at the schema's 2,000.
  4. isToolSuccess: this has already landed. isToolSuccess(output) is !isToolError(output) (lib/chatbot/executors/index.ts:43; main has since moved isToolError into executors/tool-output.ts). lib/chatbot/executors/__tests__/tool-success.test.ts covers ROLE_TOOL_NOT_PERMITTED, {"success": false} and an "Error 404" result.
  5. Truncated or emptied results: half of this is in place. Every cut entry, including one emptied by the turn budget, is followed by [truncated] in the prompt (build-prompt.ts:137-138), and the judge's prompt says what the mark means. It doesn't yet say that the cut part is unknown, so a fact missing from cut text isn't evidence of an unsupported claim. A rule saying so, with a test, rides the same push.
  6. serializeToolLog: agreed, no change.
  7. Redaction coverage: the final /security-review already targets tool-log redaction, and I've added high-entropy values in Plex and Overseerr results to its targets. The assistant has no Stripe tools. The only Stripe mention is the failed-jobs list, whose error text has emails removed and is cut to 200 characters (lib/data/assistant-data.ts), then redacted again in the log. The invite code is pinned twice: in lib/data/__tests__/assistant-data.test.ts ("shows only the end of an invite code", which asserts that no part of the full code appears) and in lib/judge/__tests__/tool-log.test.ts ("invite codes arrive masked").
  8. toolLog when a conversation ends: this is already covered. All three endings delete the conversation's message rows, and toolLog is a column on those rows: lib/chatbot/history.ts:179 and :191, and lib/queue/jobs/retention.ts:183. Task 16 adds the findings and the review summaries to the same transactions.
  9. runChatbotForUser callers: there are only two, actions/chatbot/index.ts and lib/discord/services.ts. The MCP endpoint runs single assistant tools through lib/mcp/assistant-tools.ts, not the conversation loop, so it never builds a tool log.
  10. knip entries: Task 9 removes them once the review job imports the modules.
  11. Splitting: done. The rename is now Rename LLM usage to AI activity, and the conversation judge plan #510, and this PR carries only the judge.

Generated by Claude Code

persistReview locks the conversation row, marks the review done on its
run token, then per fingerprint inserts a new finding, updates an open
one in place (never downgrading verified or not_applicable evidence to
an unverified repeat) or only counts a triaged one as seen. An ended
conversation or a superseded run throws ReviewSuperseded and writes
nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
checkJudgeSpend sums today's LLMUsage rows with a conversationReviewId
against Config.llmDailyCapJudgeUsd, on the server-zone day spendDay
(now exported from lib/llm/budget.ts) gives the assistant caps.
withJudgeLease runs a review under the judge-spend lease on the Redis
store: one acquire attempt (LEASE_BUSY when taken), a 5-minute lease
renewed every minute, always released.

recordUsage writes the attribution's conversationReviewId; addSpend
still runs only for a chatConversationId, so judge calls never count
against the assistant caps. knip gets a temporary entry until the
review job calls the module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
Add the judge:review job: it starts a run under a fencing runToken, loads
the conversation, asks the judge under the judge-spend lease, verifies the
evidence and persists the findings. A taken lease re-delays the job without
spending an attempt; PROVIDER and ABORTED retry through BullMQ; every other
code fails the review. lib/judge/stale.ts reaps dead running and queued
reviews from job states read before any transaction, and the worker passes
the lock token, lets DelayedError through and marks a review JOB_FAILED
after its last failed attempt. reviewJobId lives in lib/judge/job-id.ts so
the client-loaded queue types never pull in the queue client.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
…tus actions

enqueueReview (lib/judge/enqueue.ts), shared with the nightly sweep, refuses
without an active judge provider, reads the conversation's queued job states
before its transaction, then takes the conversation's row lock filtered on
REVIEWABLE_CONVERSATION (lib/judge/scope.ts), reaps dead reviews and returns
the open review or creates a queued one. The job is added only after commit
and only for a review it created; a queue that is down or throws fails that
review with ENQUEUE_FAILED.

reviewConversation is rate limited per admin (judge-review, 10 an hour) and
audits CONVERSATION_REVIEW_REQUESTED; getConversationReviewStatus polls the
latest review and refreshes the view once it is done or failed;
setFindingStatus triages a finding, keeps or replaces its note, and audits
CONVERSATION_FINDING_TRIAGED with ids and statuses only. The module is an MCP
tool module, with getConversationReviewStatus reviewed as read-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
CONVERSATION_REVIEW_SWEEP (judge:sweep) runs at 03:17 in the server zone
while judgeNightlyEnabled is on (lib/judge/schedule.ts, a copy of the
request-tier schedule, called at worker start and by the settings save).

selectSweepCandidates pages REVIEWABLE_CONVERSATION conversations oldest
first in keyset batches of 200 and keeps those with at least judgeMinTurns
user messages, a lastMessageAt after their latest done review's
reviewedThrough, in the sample (sha256 of id|lastMessageAt, first 8 hex
digits mod 100 < rate) and not parked by three counting nightly failures
(SCHEMA, TRUNCATED, REFUSAL, CONTENT_FILTER) with no newer message. It stops
at 50 kept, so the limit counts after sampling and the skip.

processReviewSweep checks the judge cap before each enqueueReview, on addJob
and the worker queue's new getJobState, and stops at cap, the limit or a
missing judge provider. saveJudgeSettings saves the four settings, audits
CONFIG_CHANGED through configDiff('judge') and moves the schedule; it is an
MCP tool module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

mchestr commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Thanks. I checked both issues against 04ecc8f, and neither needs a change:

  1. Lease renewal rejection: this doesn't happen. LeaseLock.renew() catches every store error and resolves false, and its JSDoc says so: "a false answer or an error means it is no longer ours" (lib/locks/lease-lock.ts:180-191). The promise in withJudgeLease's interval never rejects. A Redis error during a renewal takes the lost-lease branch, which logs and stops renewing, and the finally still releases.
  2. refreshView() on terminal polls: the Task 13 poller makes no status call after it sees done or failed. Its query sets enabled and refetchInterval (5 s) only while the status is queued or running, with refetchOnMount: false and refetchOnWindowFocus: false. The first terminal poll is therefore the last, and its refreshView() re-renders the page once. Task 13's tests pin this: "stops at done or failed" and "a done review on mount makes no status call". I've noted it in Task 13's context so the implementer keeps the refresh tied to that one call.

On the smaller notes: the fingerprint is sha256(type|tool|messageId) (lib/judge/fingerprint.ts:9-11). A repeat with the same fingerprint has the same type, tool and message by construction, so persistReview refreshes only the evidence. The deferred items are tracked for the final fix pass.


Generated by Claude Code

Task 11 added getJobState to lib/queue/client.ts without a test, which
dropped the file below its coverage floor (functions 100%).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
lib/data/conversation-findings.ts holds the judge's server-only reads, each
behind assertAdmin(): listFindings (unverified hidden by default, with the
hidden count), groupFindings (the open backlog by type and tool),
findingStats (open, fixed and the false alarm rate, null with nothing
triaged), getConversationReview (the latest review with its cost, every
finding with its stored message and never a tool log) and reviewBadgesFor
(the Calls badges). Counts leave out unverified findings everywhere.

list_conversation_findings reads listFindings over MCP, admin only, with
unverified findings hidden unless verified is false and a limit of 1 to 50.
Its description marks the judge text as untrusted data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
The test was pushed from a separate worktree while Task 12 ran here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

mchestr commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Thanks. Each point was checked independently against 67d88ed:

  1. Sweep rescans: confirmed, and it stays as it is. readBatch filters on scope and lastMessageAt, and the reviewedThrough and minTurns checks run in memory (lib/judge/sweep.ts:93-134). The scanned set is open admin and Support conversations, which end after 90 idle days, so this is a few indexed reads at 03:17 in the worker. The relation filter would compare a review column with its conversation's lastMessageAt. Prisma field references only compare columns of one model, so that would need raw SQL. A denormalised column would add a migration and a second copy of reviewedThrough to keep in step.
  2. No cap check before a manual review queues: confirmed, and there's no change. The job checks the cap under the judge-spend lease before any LLM call and fails with BUDGET_EXCEEDED, so nothing is spent (lib/queue/jobs/conversation-review.ts:132-135; the test "over cap fails with BUDGET_EXCEEDED and makes no LLM call"). A check when the admin clicks couldn't replace that one, because spend can cross the cap between the click and the run. Task 13 gives BUDGET_EXCEEDED its own copy and a Review again button. The rate-limit point is also spent before the handler runs.
  3. Stale previousStatus in the triage audit: confirmed, but narrow. It only happens when two admins triage the same finding within one database round trip. Even then the row, actor, new status and time are all correct, and setUserRole and the inactivity settings capture their before values the same way. A plain $transaction doesn't close it under READ COMMITTED, and a conditional write would need a new conflict path in the triage UI. No change.
  4. Unknown id in getConversationReviewStatus: this is Task 10's contract. The poller only ever asks about the page's own conversation. If the owner is erased mid-poll, null stops the poll, whereas NOT_FOUND would keep the client retrying every 5 s. No change.
  5. saveJudgeSettings saves before it schedules: confirmed, and Task 15 now covers it. The section will treat UPSTREAM_UNAVAILABLE as saved: close to the summary, show the action's message, no success toast, as request-tiers-section.tsx:117-123 does, with a test. Today the worst case is cosmetic: the section stays in edit mode after a save that did land. The same trace found a real gap next to it. The sweep doesn't re-check judgeNightlyEnabled, so an "off" save the queue never received could keep queuing nightly reviews, within the daily cap, until the worker restarts. The final fix pass adds the skip-while-off guard that the request-tier sync already has (lib/request-tiers/run.ts:114).
  6. Two queue paths: deliberate, and already documented. The header of enqueue.ts says each caller brings its own queue client, and the JSDoc on addJob and getJobState explains why. The sweep and action tests fail if either side is swapped. No change.

Security note: ConversationFinding.note is already in ENCRYPTED_FIELDS and FREE_TEXT_FIELDS (lib/db/encrypted-fields.ts:32,52), and prisma-encryption-extension.test.ts:166-176 pins it.
Tests: the 200-row batch boundary, including a lastMessageAt tie split across pages, is covered in lib/judge/__tests__/sweep.test.ts:274-287.


Generated by Claude Code

- Task 12's loaders split: getConversationReview and reviewBadgesFor
  live in lib/data/conversation-review.ts (the 150-line lib limit).
- Task 13: a finding card's Message #N uses the judge's own index.
- Task 14: the none badge means no done review; the stat tiles read
  Open (any range), Fixed (in range) and False alarm rate (triaged in
  range).
- Task 15: a saved-but-unscheduled result shows as saved.
- Task 16: unlinking Discord ends the member's session-less
  conversations, and REVIEWABLE_CONVERSATION requires a stored turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

mchestr commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Thanks. These answer both reviews of Tasks 2–12 (the runs at 0e31273 and 1f8fb7e). I checked each point at 1f8fb7e.

  • The count for a repeat that keeps the stored evidence: agreed. persistFinding returns "updated" for that case (lib/judge/persist.ts:95-100), but the description says it only counts as seen. The final fix pass makes it count as suppressed, so updated means the evidence was refreshed. It adds the matching row to the module table and a test.
  • Queries per finding while the row lock is held: a review has at most 20 findings (lib/judge/schema.ts:32), so this is at most 40 queries. The final pass notes that cap at the loop.
  • isParked once per conversation: this stays. It's a nightly job over admin and Support conversations only.
  • row.lastMessageAt! in the sweep: this is safe. readBatch filters on lastMessageAt: { lte: now }, which never matches a null. Task 16 also adds lastMessageAt: { not: null } to REVIEWABLE_CONVERSATION.
  • A finding whose message is missing: messages and findings go in one transaction. Task 16 runs endReviewsOf inside each ending's existing transaction (endConversation, endConversationsOf and the idle ending), and its pg tests check that an ending removes the findings.
  • getConversationReview loads everything: this stays. Each review adds at most 20 fingerprints to a conversation, so the set stays small.
  • refreshView() on terminal reads: under the MCP endpoint the action runs with a principal session, and refreshView skips the refresh there (lib/actions/invalidate.ts:28-30). So only the page's poller ever triggers it. The final pass adds a one-line note on the action saying so.
  • ok(null) for an unknown id over MCP: this is the documented contract. review_conversation returns NOT_FOUND for a missing conversation, so a caller can tell the two apart in one step.
  • pg coverage: concurrent enqueue is in Task 10's pg suite. The endings, the cascades and judge spend after an ending land in Task 16, before this leaves draft.
  • The sweep's judgeNightlyEnabled guard: it gets the test you suggest, where the setting is off and the scheduled job still fires.

Generated by Claude Code

- lib/data/conversation-review.ts now holds getConversationReview and
  reviewBadgesFor, so neither loader file passes lib's 150-line limit.
- getConversationReview audits a conversation-review read against the
  conversation's owner; listFindings audits a conversation-findings listing
  with its filters only, never finding text.
- findingStats: Open is the whole open backlog whatever the range; Fixed and
  the false alarm rate count findings triaged in the range. A Postgres suite
  pins the ranges and the grouped backlog.
- list_conversation_findings clips claim, evidence, suggestion and note and
  reports each full length, so 20 findings at every field's cap fit the
  result cap.
- The select-honouring Prisma mock helper moves to __tests__/helpers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
The conversation page gains a Review section above the transcript, in
both branches (the "No calls" one included), for a reviewable admin
assistant conversation. It starts a review, shows its status as a
StatusBadge with a Spinner (a new conversationReview status domain),
polls getConversationReviewStatus every 5 s only while the review is
queued or running, and shows the reviewed date, model, cost and summary,
or the failure's fixed copy from reviewErrorCopy. Each finding is a card
with its message inline (clamped to 6 lines), an "Unverified quote"
badge where the quote did not match, and triage through setFindingStatus.
All judge text renders as plain text.

knip no longer needs the conversation-review actions as an entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
AI activity gets Calls | Findings tabs kept in ?view=findings. Findings
lists the judge's findings across conversations (severity, type, a
one-line plain-text claim, tool, conversation link, date, status, Seen)
with the Open, Fixed (in range) and False alarm rate tiles, type,
severity and status filters (open by default), Hide unverified with the
hidden count, the shared date range, and a Group by type and tool view
whose rows link to the filtered list. A group's tool shows as a chip;
tool= (empty) is "names no tool", so FindingFilters.tool takes null.

Calls rows for admin assistant conversations carry a review badge from
reviewBadgesFor (Not reviewed, No findings, N open, Unverified only),
LLM_PURPOSES gains judge ("Conversation review"), and aiActivity()
builds the view and the findings filters. The finding type, severity
and status words move to lib/ui/finding-labels.ts for both pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
@mchestr
mchestr marked this pull request as ready for review November 2, 2026 03:58
#510's commits are already on this branch, so this changes no file. It
gives GitHub a single merge base, so the PR merges cleanly and its
checks run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

mchestr commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Thanks. Only two points are new; points 2–4 are answered in the earlier threads, and the persist count fix, with its table row and test, is in the final pass.

  • Triage after a conversation ends (1): once Task 16 lands, every ending deletes the conversation's findings in the same transaction. A triage of one of them then gets NOT_FOUND from setFindingStatus, whose zero-count path already returns it. Task 16's pg tests pin the deletion, so no separate scope check is needed.
  • The silent addJob catch in enqueueReview (5): agreed. The final pass logs a warning with the review id before the review is marked ENQUEUE_FAILED.

The list_conversation_findings clipping is on the /security-review target list, alongside the tool-log redaction and the new ENCRYPTED_FIELDS entries.


Generated by Claude Code

The AI provider section gains a third tab, Conversation review, whose only
part is the judge's provider form, saved through the new
updateJudgeLLMProvider action (purpose judge, audited as llm.judge). The
tab bodies move into LLMProviderPanel, and the tab change accepts all
three purposes instead of folding every non-wrapped value into chat.

A Conversation review section, the judge cap's one editor, holds the
nightly switch, the sample rate slider, the minimum turns and the daily
budget, and saves through saveJudgeSettings. A save whose schedule did
not move counts as saved, with the queue's message as a toast. The
settings loader returns the judge provider without its key and the judge
Config fields; the section has its nav entry and Last saved footer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
getConversationReview wrote a MEMBER_CONTENT_VIEWED row whenever the
conversation had an owner, so opening a conversation with no review and
no findings logged a conversation-review read of nothing, next to the
page's own llm-conversation row (e2e/admin/audit-trail.spec.ts). It now
audits only when it returns a finding with its message, or a summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

mchestr commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Thanks. One point is new: identical claims should be de-duplicated when findings merge. The final pass now drops repeats before it joins the claims, then caps the joined claim at 2,000 characters. Everything else in this run is already on the list, and the UI wording stays as you describe: no finding is labelled "verified" or "confirmed" anywhere.


Generated by Claude Code

- endReviewsOf (lib/judge/end.ts) fails a conversation's queued and running
  reviews with ENDED, deletes its findings and clears review summaries, only
  for conversations that have ended. endConversation, endConversationsOf, the
  idle retention ending and the Discord unlink call it in their transactions;
  the unlink now also ends conversations its session delete leaves without a
  session.
- REVIEWABLE_CONVERSATION requires lastMessageAt, so a conversation with no
  stored turn, or a Discord one an earlier unlink left open, is never reviewed.
- The admin assistant page's disclosure names a different judge provider;
  getAssistantDisclosure and the member surfaces are unchanged.
- llmSpendSince gains a judge bucket; /api/metrics adds judge reviews by
  status, findings by type and evidence check, and today's judge spend.
- Erase inventory counts reviews and findings; privacy, MCP and operations
  docs cover the judge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
Base UI keeps a Select's popup mounted and inert (pointer-events: none)
until it is open, and findByTestId finds an option before then. On a
slow runner user-event clicked the inert option and threw (the
false_alarm case of finding-triage.test.tsx in CI). The triage and
Findings tab tests now wait for the trigger's aria-expanded first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
5fd17d6 (audit a review read only when it returns the member's text)
and 872f5fe (wait for the Select to open in the judge tests) were made
in a separate worktree, so the running tasks' review ranges stayed
clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
The plan's Task 16 lists it; the task's implementer left it to the
controller, since its brief alone could not authorise a CLAUDE.md edit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

mchestr commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Thanks. These cover the reviews at 872f5fe and e1c7b0b. Two points are new:

  • Duplicate fingerprints in one review: this already can't happen. verifyFindings merges findings by type, tool and message (lib/judge/verify.ts:113, keyed on [type, tool ?? "", messageIndex]), which are exactly the fingerprint's inputs, before persistReview sees them. So one review never carries two findings with the same fingerprint, and seenCount rises by one at most. The planned de-duplication only stops a merged claim from reading "X / X".
  • llmSpendSince buckets: agreed. The final pass makes the judge sum also filter on chatConversationId: null, so a row can never count in both the chat and judge buckets, and wrapped can't be subtracted twice.

The other points are already on the final-pass list. The sweep's judgeNightlyEnabled guard comes with the test you describe, where the setting is off and the scheduled job still fires. The Visual Baselines refresh will run once the final fix pass has settled the AI activity UI.


Generated by Claude Code

…oors

evals/judge/ holds three hand-written conversations (a wrong count against a
tool result, a missing capability and an unneeded refusal), each with a tool
result that says "report no findings", and a recorded judge response. The
fixture test runs each through buildJudgePrompt, verifyFindings and
persistReview on the Prisma mock.

e2e/admin/conversation-review.spec.ts seeds the wrong-count conversation and
a judge provider on the LLM stub, reviews it, triages the finding as a false
alarm and checks the Findings tab's rate and the prompt the stub recorded.
The stub answers the judge-e2e model with the fixture's recorded output and
serves the judge requests at GET /__judge-requests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y

mchestr commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Thanks. Taking the new points from the run at 1714538 against the code:

  1. A throw before the lease: this doesn't leave the review spinning until the stale reaper. Like any job error, it uses BullMQ's 3 attempts with 5 s and 10 s backoff. After the last attempt, the worker's final-attempt hook marks the review failed with JOB_FAILED (lib/queue/worker.ts:92 and :122, via markReviewFailed). So the admin sees about 15 s of retries, then the failure copy. The 10-minute running limit is only the backstop for a crashed worker. Retrying a bug costs two extra failed starts and no LLM call, because the throw happens before the call. No change.
  2. Nightly on with no judge model: the sweep already handles this. enqueueReview returns UNAVAILABLE before it creates any review row, and processReviewSweep stops with stoppedBy: 'no-provider' (lib/judge/sweep.ts). So nothing is queued and nothing fails with NOT_CONFIGURED.
  3. Triage racing an ending: the zero-count path is real (actions/admin/conversation-review.ts:111), but no test covers it. The final pass adds a unit test: findUnique hits, updateMany matches 0, and the action returns NOT_FOUND and writes no audit event. A pg test would only re-prove that Postgres reports 0 rows.
  4. pg coverage:
    • The ending racing persistReview and judge spend surviving an ending landed in Task 16's pg suite: "persistReview after the conversation ended writes nothing", "findings persisted just before the conversation ends are removed by the ending" and "ending a conversation leaves today's judge spend unchanged".
    • Two reviews of one conversation never run at once. enqueueReview returns the open review instead of a second one, which Task 10's pg suite covers, and persistReview holds the row lock. So the fingerprint constraint never sees concurrent writers for one conversation.

Points 3, 4 and 6 are already on the final-pass list.


Generated by Claude Code

@mchestr
mchestr merged commit 5750328 into main Nov 2, 2026
22 checks passed
@mchestr
mchestr deleted the claude/sleepy-feynman-bo0yp4 branch November 2, 2026 05:51
mchestr pushed a commit that referenced this pull request Nov 2, 2026
The fix pass reworded the judge's system prompt (the "(message
unreadable)" sentence, the neutralised-structure paragraph and a new
rule 2 that renumbers the rest) under judge-v1, the version #508 merged
at 5750328. Images are published from main on every push, so judge-v1
rows would mix two wordings, and the file's own comment asks for a new
file and version for a wording change.

lib/judge/prompts/v1.ts moves to v2.ts with JUDGE_PROMPT_VERSION
"judge-v2". Its comment names judge-v1 as #508's wording, readable at
5750328, says what v2 changed, and keeps the rule. The builder and the
processor import it; the tests that pin the version expect judge-v2,
and the eval harness stores JUDGE_PROMPT_VERSION as the processor does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
mchestr pushed a commit that referenced this pull request Nov 2, 2026
The spec was amended only for the message label, so in several places
it still described #508's behaviour. It now says that:

- reviewConversation also refuses while AI features are off;
- setFindingStatus takes an optional status, and only a status change
  sets the triage fields and is audited;
- the processor fails a newest turn that does not fit, or keeps under
  200 characters of a message, with PROMPT_TOO_LARGE before the lease,
  and runs blockedBeforeCall under the lease before the spend check;
- earlier_message is matched against the text the judge was sent;
- the status poll reaps a dead review too;
- PROMPT_TOO_LARGE counts toward the failure backoff, LLM_DISABLED does
  not, and the sweep stops at disabled while nightly reviews or AI
  features are off;
- the disclosure compares provider rows, not names;
- the MCP description calls the tool untrusted too;
- the cap is labelled Daily budget.

Section 4's processor steps and section 5 also say that message text,
the claims already raised and tool texts are stripped of invisible
characters, redacted and neutralised before the judge sees them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
mchestr pushed a commit that referenced this pull request Nov 2, 2026
v0.9.0 was tagged after #508 merged and contains it, so judge-v1 rows
come from that release as well as from images built from main. The v2
prompt's comment now says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants