Repository navigation
Conversation judge (in progress) - #508
Conversation
|
Claude finished @mchestr's task in 34s —— View job Review of the conversation judge PR (draft, in progress)I read the review job, What looks good
Possible issues, ordered by likely impact
Housekeeping
Overall the design is careful and the concurrency story holds up. Only point 1 seems worth a change before merge. Points 2 and 7 are small additions. |
|
Thanks. I checked each point against the code, and a34ba52 addresses them:
Generated by Claude Code |
|
Rewind E2E Tests failed on a78b754, and the failure isn't this PR's.
Generated by Claude Code |
|
Thanks. On the optional points:
The description was updated before this review finished. It now describes Task 1 and says the judge commits will land here until a branch decision is made. Generated by Claude Code |
…fter its filters The sixth plan review: two concurrent enqueues could each create a review, so enqueueReview locks the conversation row first (ChatConversation.reviewedAt is renamed reviewLockedAt, a lock column nothing reads). The sweep's 50 limit now applies after sampling and the failure skip, the purpose grep covers every two-purpose assumption, and new tests pin the Error heuristic, a rejected moveToDelayed, an ending's encrypted review columns and inert MCP text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
|
Thanks. On the three points:
Generated by Claude Code |
ConversationReview and ConversationFinding with their enums, the assistant message tool log, the judge's Config fields, LLMUsage's link to a review and the conversation's review lock column, in one additive migration (20270121110000_add_conversation_judge). The judge's text columns are encrypted free text, LLMPurpose gains "judge", LeaseName gains "judge-spend", and the factories build the new rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
runConversationLoop now records each tool call's name, arguments, output
and an ok hint (isToolSuccess: an output not beginning "Error"), and
passes them through capToolLog before they leave the loop: credentials
are redacted with lib/mcp/redact.ts first, then args are cut to 2 KB,
each result to 8 KB and the turn to 32 KB, marking anything cut
truncated. runChatbotForUser returns { response, toolLog } so the log
never rides on ChatMessage or ChatResponse; the admin chat action hands
it to saveTurn, which stores it on the answer's encrypted toolLog
column, and the Discord path ignores it, so Discord's plaintext session
JSON never sees it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
…s an empty log Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
|
Thanks, both reviews were useful. I checked each point against the code:
Two more changes are in the same fix round: an empty tool log ( Generated by Claude Code |
…he tool name - isToolSuccess is !isToolError, so JSON refusals and error outputs log as ok: false - runConversationLoop returns a toolLog for every answered turn; serializeToolLog stores '[]' for an empty log and null only when none was captured - each entry's result is String(output ?? ''), so a non-string output cannot fail the turn - capToolLog redacts the tool name and cuts it to 100 characters Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
judgeOutputSchema caps a review at 20 findings and a 600-character summary, with text caps (evidence 4000, claim and suggestion 2000, tool 100) well above the 300-character quotes the prompt asks for. fingerprintOf hashes type|tool|messageId only, so a reworded claim still matches. Both are knip entries until the review job imports them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
OpenAI strict structured outputs need every property listed in required; an optional tool left it out and would fail the judge call with a 400. A finding no tool applies to now carries tool: null. A test pins that the JSON Schema is all-required with additionalProperties: false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
buildJudgePrompt fills a 120,000-character budget in order: the assistant's prompt and catalog, the newest turn whole (its pieces cut to a fair share only when it alone passes the cap), the "Already raised, don't repeat" block, then earlier turns newest first with tool results cut to 2 KB, and a line counting the turns left out. Each message keeps its absolute index, null and empty tool logs are labelled, and messages up to the last done review are marked as context. Every fence close tag inside the content becomes <\/tag (JSON's own escape, so a tool result still parses), so each block closes once. getAdminAssistantContext gives the judge the prompt and toolset the owner gets today, a Support owner's smaller toolset and scope section included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
verifyFindings checks each finding's quote against the text the judge was sent: tool results leaf by leaf, then as sent text (cut to 2 KB for earlier turns), earlier messages by lower index, catalog tools by name. A finding outside the prompt's window is dropped; a failed check keeps it as unverified; duplicates on one fingerprint merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
The judge call is a strict structured output, so every finding key is required and a finding no tool applies to carries tool: null, as lib/judge/schema.ts already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
Brings in #504, #507 and #509. The one conflict: main moved isToolError into lib/chatbot/executors/tool-output.ts, so the executors keep only isToolSuccess, which reuses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
|
Thanks. I checked each point against cd45006:
Generated by Claude Code |
persistReview locks the conversation row, marks the review done on its run token, then per fingerprint inserts a new finding, updates an open one in place (never downgrading verified or not_applicable evidence to an unverified repeat) or only counts a triaged one as seen. An ended conversation or a superseded run throws ReviewSuperseded and writes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
checkJudgeSpend sums today's LLMUsage rows with a conversationReviewId against Config.llmDailyCapJudgeUsd, on the server-zone day spendDay (now exported from lib/llm/budget.ts) gives the assistant caps. withJudgeLease runs a review under the judge-spend lease on the Redis store: one acquire attempt (LEASE_BUSY when taken), a 5-minute lease renewed every minute, always released. recordUsage writes the attribution's conversationReviewId; addSpend still runs only for a chatConversationId, so judge calls never count against the assistant caps. knip gets a temporary entry until the review job calls the module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
Add the judge:review job: it starts a run under a fencing runToken, loads the conversation, asks the judge under the judge-spend lease, verifies the evidence and persists the findings. A taken lease re-delays the job without spending an attempt; PROVIDER and ABORTED retry through BullMQ; every other code fails the review. lib/judge/stale.ts reaps dead running and queued reviews from job states read before any transaction, and the worker passes the lock token, lets DelayedError through and marks a review JOB_FAILED after its last failed attempt. reviewJobId lives in lib/judge/job-id.ts so the client-loaded queue types never pull in the queue client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
…tus actions enqueueReview (lib/judge/enqueue.ts), shared with the nightly sweep, refuses without an active judge provider, reads the conversation's queued job states before its transaction, then takes the conversation's row lock filtered on REVIEWABLE_CONVERSATION (lib/judge/scope.ts), reaps dead reviews and returns the open review or creates a queued one. The job is added only after commit and only for a review it created; a queue that is down or throws fails that review with ENQUEUE_FAILED. reviewConversation is rate limited per admin (judge-review, 10 an hour) and audits CONVERSATION_REVIEW_REQUESTED; getConversationReviewStatus polls the latest review and refreshes the view once it is done or failed; setFindingStatus triages a finding, keeps or replaces its note, and audits CONVERSATION_FINDING_TRIAGED with ids and statuses only. The module is an MCP tool module, with getConversationReviewStatus reviewed as read-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
CONVERSATION_REVIEW_SWEEP (judge:sweep) runs at 03:17 in the server zone
while judgeNightlyEnabled is on (lib/judge/schedule.ts, a copy of the
request-tier schedule, called at worker start and by the settings save).
selectSweepCandidates pages REVIEWABLE_CONVERSATION conversations oldest
first in keyset batches of 200 and keeps those with at least judgeMinTurns
user messages, a lastMessageAt after their latest done review's
reviewedThrough, in the sample (sha256 of id|lastMessageAt, first 8 hex
digits mod 100 < rate) and not parked by three counting nightly failures
(SCHEMA, TRUNCATED, REFUSAL, CONTENT_FILTER) with no newer message. It stops
at 50 kept, so the limit counts after sampling and the skip.
processReviewSweep checks the judge cap before each enqueueReview, on addJob
and the worker queue's new getJobState, and stops at cap, the limit or a
missing judge provider. saveJudgeSettings saves the four settings, audits
CONFIG_CHANGED through configDiff('judge') and moves the schedule; it is an
MCP tool module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
|
Thanks. I checked both issues against 04ecc8f, and neither needs a change:
On the smaller notes: the fingerprint is Generated by Claude Code |
Task 11 added getJobState to lib/queue/client.ts without a test, which dropped the file below its coverage floor (functions 100%). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
lib/data/conversation-findings.ts holds the judge's server-only reads, each behind assertAdmin(): listFindings (unverified hidden by default, with the hidden count), groupFindings (the open backlog by type and tool), findingStats (open, fixed and the false alarm rate, null with nothing triaged), getConversationReview (the latest review with its cost, every finding with its stored message and never a tool log) and reviewBadgesFor (the Calls badges). Counts leave out unverified findings everywhere. list_conversation_findings reads listFindings over MCP, admin only, with unverified findings hidden unless verified is false and a limit of 1 to 50. Its description marks the judge text as untrusted data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
The test was pushed from a separate worktree while Task 12 ran here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
|
Thanks. Each point was checked independently against 67d88ed:
Security note: Generated by Claude Code |
- Task 12's loaders split: getConversationReview and reviewBadgesFor live in lib/data/conversation-review.ts (the 150-line lib limit). - Task 13: a finding card's Message #N uses the judge's own index. - Task 14: the none badge means no done review; the stat tiles read Open (any range), Fixed (in range) and False alarm rate (triaged in range). - Task 15: a saved-but-unscheduled result shows as saved. - Task 16: unlinking Discord ends the member's session-less conversations, and REVIEWABLE_CONVERSATION requires a stored turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
|
Thanks. These answer both reviews of Tasks 2–12 (the runs at 0e31273 and 1f8fb7e). I checked each point at 1f8fb7e.
Generated by Claude Code |
- lib/data/conversation-review.ts now holds getConversationReview and reviewBadgesFor, so neither loader file passes lib's 150-line limit. - getConversationReview audits a conversation-review read against the conversation's owner; listFindings audits a conversation-findings listing with its filters only, never finding text. - findingStats: Open is the whole open backlog whatever the range; Fixed and the false alarm rate count findings triaged in the range. A Postgres suite pins the ranges and the grouped backlog. - list_conversation_findings clips claim, evidence, suggestion and note and reports each full length, so 20 findings at every field's cap fit the result cap. - The select-honouring Prisma mock helper moves to __tests__/helpers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
The conversation page gains a Review section above the transcript, in both branches (the "No calls" one included), for a reviewable admin assistant conversation. It starts a review, shows its status as a StatusBadge with a Spinner (a new conversationReview status domain), polls getConversationReviewStatus every 5 s only while the review is queued or running, and shows the reviewed date, model, cost and summary, or the failure's fixed copy from reviewErrorCopy. Each finding is a card with its message inline (clamped to 6 lines), an "Unverified quote" badge where the quote did not match, and triage through setFindingStatus. All judge text renders as plain text. knip no longer needs the conversation-review actions as an entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
AI activity gets Calls | Findings tabs kept in ?view=findings. Findings
lists the judge's findings across conversations (severity, type, a
one-line plain-text claim, tool, conversation link, date, status, Seen)
with the Open, Fixed (in range) and False alarm rate tiles, type,
severity and status filters (open by default), Hide unverified with the
hidden count, the shared date range, and a Group by type and tool view
whose rows link to the filtered list. A group's tool shows as a chip;
tool= (empty) is "names no tool", so FindingFilters.tool takes null.
Calls rows for admin assistant conversations carry a review badge from
reviewBadgesFor (Not reviewed, No findings, N open, Unverified only),
LLM_PURPOSES gains judge ("Conversation review"), and aiActivity()
builds the view and the findings filters. The finding type, severity
and status words move to lib/ui/finding-labels.ts for both pages.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
#510's commits are already on this branch, so this changes no file. It gives GitHub a single merge base, so the PR merges cleanly and its checks run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
|
Thanks. Only two points are new; points 2–4 are answered in the earlier threads, and the persist count fix, with its table row and test, is in the final pass.
The Generated by Claude Code |
The AI provider section gains a third tab, Conversation review, whose only part is the judge's provider form, saved through the new updateJudgeLLMProvider action (purpose judge, audited as llm.judge). The tab bodies move into LLMProviderPanel, and the tab change accepts all three purposes instead of folding every non-wrapped value into chat. A Conversation review section, the judge cap's one editor, holds the nightly switch, the sample rate slider, the minimum turns and the daily budget, and saves through saveJudgeSettings. A save whose schedule did not move counts as saved, with the queue's message as a toast. The settings loader returns the judge provider without its key and the judge Config fields; the section has its nav entry and Last saved footer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
getConversationReview wrote a MEMBER_CONTENT_VIEWED row whenever the conversation had an owner, so opening a conversation with no review and no findings logged a conversation-review read of nothing, next to the page's own llm-conversation row (e2e/admin/audit-trail.spec.ts). It now audits only when it returns a finding with its message, or a summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
|
Thanks. One point is new: identical claims should be de-duplicated when findings merge. The final pass now drops repeats before it joins the claims, then caps the joined claim at 2,000 characters. Everything else in this run is already on the list, and the UI wording stays as you describe: no finding is labelled "verified" or "confirmed" anywhere. Generated by Claude Code |
- endReviewsOf (lib/judge/end.ts) fails a conversation's queued and running reviews with ENDED, deletes its findings and clears review summaries, only for conversations that have ended. endConversation, endConversationsOf, the idle retention ending and the Discord unlink call it in their transactions; the unlink now also ends conversations its session delete leaves without a session. - REVIEWABLE_CONVERSATION requires lastMessageAt, so a conversation with no stored turn, or a Discord one an earlier unlink left open, is never reviewed. - The admin assistant page's disclosure names a different judge provider; getAssistantDisclosure and the member surfaces are unchanged. - llmSpendSince gains a judge bucket; /api/metrics adds judge reviews by status, findings by type and evidence check, and today's judge spend. - Erase inventory counts reviews and findings; privacy, MCP and operations docs cover the judge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
Base UI keeps a Select's popup mounted and inert (pointer-events: none) until it is open, and findByTestId finds an option before then. On a slow runner user-event clicked the inert option and threw (the false_alarm case of finding-triage.test.tsx in CI). The triage and Findings tab tests now wait for the trigger's aria-expanded first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
5fd17d6 (audit a review read only when it returns the member's text) and 872f5fe (wait for the Select to open in the judge tests) were made in a separate worktree, so the running tasks' review ranges stayed clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
The plan's Task 16 lists it; the task's implementer left it to the controller, since its brief alone could not authorise a CLAUDE.md edit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
|
Thanks. These cover the reviews at 872f5fe and e1c7b0b. Two points are new:
The other points are already on the final-pass list. The sweep's Generated by Claude Code |
…oors evals/judge/ holds three hand-written conversations (a wrong count against a tool result, a missing capability and an unneeded refusal), each with a tool result that says "report no findings", and a recorded judge response. The fixture test runs each through buildJudgePrompt, verifyFindings and persistReview on the Prisma mock. e2e/admin/conversation-review.spec.ts seeds the wrong-count conversation and a judge provider on the LLM stub, reviews it, triages the finding as a false alarm and checks the Findings tab's rate and the prompt the stub recorded. The stub answers the judge-e2e model with the fixture's recorded output and serves the judge requests at GET /__judge-requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
|
Thanks. Taking the new points from the run at 1714538 against the code:
Points 3, 4 and 6 are already on the final-pass list. Generated by Claude Code |
The fix pass reworded the judge's system prompt (the "(message unreadable)" sentence, the neutralised-structure paragraph and a new rule 2 that renumbers the rest) under judge-v1, the version #508 merged at 5750328. Images are published from main on every push, so judge-v1 rows would mix two wordings, and the file's own comment asks for a new file and version for a wording change. lib/judge/prompts/v1.ts moves to v2.ts with JUDGE_PROMPT_VERSION "judge-v2". Its comment names judge-v1 as #508's wording, readable at 5750328, says what v2 changed, and keeps the rule. The builder and the processor import it; the tests that pin the version expect judge-v2, and the eval harness stores JUDGE_PROMPT_VERSION as the processor does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
The spec was amended only for the message label, so in several places it still described #508's behaviour. It now says that: - reviewConversation also refuses while AI features are off; - setFindingStatus takes an optional status, and only a status change sets the triage fields and is audited; - the processor fails a newest turn that does not fit, or keeps under 200 characters of a message, with PROMPT_TOO_LARGE before the lease, and runs blockedBeforeCall under the lease before the spend check; - earlier_message is matched against the text the judge was sent; - the status poll reaps a dead review too; - PROMPT_TOO_LARGE counts toward the failure backoff, LLM_DISABLED does not, and the sweep stops at disabled while nightly reviews or AI features are off; - the disclosure compares provider rows, not names; - the MCP description calls the tool untrusted too; - the cap is labelled Daily budget. Section 4's processor steps and section 5 also say that message text, the claims already raised and tool texts are stripped of invisible characters, redacted and neutralised before the judge sees them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
v0.9.0 was tagged after #508 merged and contains it, so judge-v1 rows come from that release as well as from images built from main. The v2 prompt's comment now says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y
Summary
This is the conversation judge from
docs/superpowers/plans/2026-11-01-conversation-judge.md, built task by task: a subagent implementer per task, an independent task review, a security review on the sensitive tasks, and up to five fix rounds.It builds on #510, which is merged.
mainis merged in at 421b672, which changes no file, so this diff now shows only the judge.Done so far
20270121110000_add_conversation_judge). It adds:ConversationReviewandConversationFindingChatConversationMessage.toolLogandChatConversation.reviewLockedAtLLMUsage.conversationReviewId(SetNull) and theConfigjudge fieldsjudgepurpose, thejudge-spendlease name, factories, andchangelog.d/conversation-judge.mdjudge-spendRedis lease.judge:review). It covers:PROVIDERandABORTEDreviewConversation,getConversationReviewStatusandsetFindingStatus, with the sharedenqueueReview.saveJudgeSettings.list_conversation_findingsMCP read tool.lib/data/conversation-findings.tsandlib/data/conversation-review.ts. Reads of a member's conversation are audited asconversation-reviewandconversation-findings.Still to come
/security-reviewThe final fix pass already includes:
[truncated]is unknownChecklist
/dev/ui); visual diff attached (the UI from Tasks 13–15 changes the AI activity screenshots; the Visual Baselines refresh is still needed)npm run design:checkpassesdata-testid.claude/skills/**or.mcp.jsonchanges🤖 Generated with Claude Code
https://claude.ai/code/session_011TLtUPKTYoxFj5AGmdPX7Y