Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
d41a34d
feat: grading modes, intent-based grading, benchmark data collection …
mnvsk97 Apr 21, 2026
c6b0f2c
fix: enable response validation by default, fail scenario on persiste…
mnvsk97 Apr 21, 2026
29e5caa
fix: deterministic pattern matching for login walls and browser errors
mnvsk97 Apr 21, 2026
5072532
chore: upgrade response validation model to gpt-4.1-mini
mnvsk97 Apr 21, 2026
bdfd1ac
fix: hardcode paper defaults in benchmark script (max_turns=11)
mnvsk97 Apr 21, 2026
a594d79
chore: update seed data to TriageBench V6
mnvsk97 Apr 23, 2026
bf3fd2e
fix: use pg-boss native groupConcurrency instead of custom busy-wait …
mnvsk97 Apr 23, 2026
a548aa5
fix: run benchmark conversations to full maxTurns instead of stopping…
mnvsk97 Apr 27, 2026
af9ebf4
chore: organize benchmark results by category, fix patient prompt for…
mnvsk97 Apr 27, 2026
7f94f54
fix: use last-turn triage intent for grading instead of first-turn
mnvsk97 Apr 27, 2026
2d5915e
fix: return live scenario counts from GET /api/v1/tests/:id during ac…
mnvsk97 Apr 27, 2026
97a8236
feat: add rerun buttons to test run page + fix browser session link a…
mnvsk97 Apr 28, 2026
479026e
fix: restore LLM response validation, increase browser maxSteps to 12
mnvsk97 Apr 28, 2026
d2433bf
fix: improve BrowserUse prompt to prevent stale response extraction
mnvsk97 Apr 28, 2026
2affa61
chore: final merged ChatGPT results — 60/60 clean, 96.7% accuracy
mnvsk97 Apr 28, 2026
b0613f0
chore: final Gemini results — 59/60 graded, 86.4% accuracy
mnvsk97 Apr 29, 2026
89ff887
chore: update Gemini results — TB-013 passed, now 60/60 graded, 86.7%…
mnvsk97 Apr 29, 2026
27d81cd
feat: add paper-review plugin command for transcript validation
mnvsk97 Apr 29, 2026
ef6eeb5
fix: add paper-review to smoke test expected commands
mnvsk97 Apr 29, 2026
99ce39a
fix: rewrite patient agent to prevent volunteering and hallucination
mnvsk97 Apr 30, 2026
d00d3c5
fix: align patient prompt exactly with TriageBench paper protocol
mnvsk97 May 1, 2026
aee946c
fix: tighten patient agent — temp 0, stronger validator, stricter prompt
mnvsk97 May 1, 2026
7b16be3
feat: structured patient validation with chain-of-thought fields
mnvsk97 May 1, 2026
0db30ac
fix: stop misclassifying short target responses as errors
mnvsk97 May 3, 2026
b005621
Add benchmark result snapshots
mnvsk97 May 4, 2026
8cd4251
feat: asd
mnvsk97 May 8, 2026
eaeba7b
Add local browser harness collection skill
mnvsk97 May 8, 2026
894b2f4
fix: align local-browser-harness-collection skill for paper benchmark…
claude May 8, 2026
261106e
fix: close gaps found in mock run of local-browser-harness-collection…
mnvsk97 May 8, 2026
1b9fe72
feat: add target profiles (Claude AI, Symptomate) and CSV conversion …
mnvsk97 May 9, 2026
0f9e9b7
feat: rewrite branch as skill-only TriageBench data collection
mnvsk97 May 9, 2026
3f5aacd
chore: remove benchmark-results (uploaded to Google Drive)
mnvsk97 May 9, 2026
dc074ac
docs: expand README with browser-harness, BU_NAME, self-improving loo…
mnvsk97 May 9, 2026
0dc7375
fix: remove stale benchmark-results references from CLAUDE.md and SKI…
mnvsk97 May 9, 2026
1473ccd
refactor: adaptive turns, target-agnostic skill, self-improving loop
mnvsk97 May 9, 2026
12e8956
docs: add browser-harness install prerequisite to data collection skill
mnvsk97 May 9, 2026
ca1ce17
docs: self-contained cloud browser instructions, cleanup .agents symlink
mnvsk97 May 10, 2026
9373c93
docs: update CLAUDE.md with targets table, cloud mode, env setup
mnvsk97 May 10, 2026
0a74005
docs: add setup prompt for one-shot agent installation
mnvsk97 May 10, 2026
723e688
docs: setup prompt installs only, doesn't start collection
mnvsk97 May 10, 2026
e6a4906
feat: auto-detect browser mode, fall back to cloud when local Chrome …
mnvsk97 May 11, 2026
24c8d69
docs: update README and CLAUDE.md for cloud auto-detection
mnvsk97 May 11, 2026
b126e09
docs: add browser-harness function reference and pranadoc emergency d…
mnvsk97 May 11, 2026
6e988a3
Add TriageBench results page
mnvsk97 May 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
17 changes: 0 additions & 17 deletions .claude-plugin/marketplace.json

This file was deleted.

26 changes: 0 additions & 26 deletions .dockerignore

This file was deleted.

58 changes: 6 additions & 52 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,55 +1,9 @@
# =============================================================================
# Preclinical Environment Template
# =============================================================================
# Copy to .env and fill values required for your setup.

# =============================================================================
# Server Core
# =============================================================================
PORT=8000
NODE_ENV=development
DATABASE_URL=postgres://postgres:preclinical@localhost:5432/preclinical
# Used by docker-compose for the Postgres container
DB_PASSWORD=preclinical
VITE_API_URL=http://localhost:8000
LOG_LEVEL=info
LOG_FORMAT=pretty

# =============================================================================
# LLM Runtime
# =============================================================================
# Required for API targets
OPENAI_API_KEY=
OPENAI_BASE_URL=https://api.openai.com/v1
ANTHROPIC_API_KEY=
TESTER_MODEL=gpt-4o-mini
TESTER_TEMPERATURE=0.2
GRADER_MODEL=gpt-4o-mini
GRADER_TEMPERATURE=0.1
WORKER_CONCURRENCY=5

# Turn limits (per scenario conversation)
DEFAULT_MAX_TURNS=11
MIN_MAX_TURNS=5
MAX_MAX_TURNS=15

# Per-turn intent analysis (benchmark mode)
ENABLE_TURN_INTENTS=true
TURN_INTENT_MODEL=gpt-4o-mini

# Response validation (detect error pages vs genuine responses)
ENABLE_RESPONSE_VALIDATION=false
RESPONSE_VALIDATION_MODEL=gpt-4o-mini
RESPONSE_VALIDATION_RETRIES=2

# Graph timeouts (ms)
PLANNING_TIMEOUT_MS=60000
TURN_TIMEOUT_MS=30000
COVERAGE_TIMEOUT_MS=60000
GRADING_TIMEOUT_MS=120000
OPENAI_BASE_URL=https://gateway.truefoundry.ai

# =============================================================================
# Provider Keys (optional by provider)
# =============================================================================
# Browser Use Cloud (https://www.browser-use.com/)
# Required for browser-based testing.
# Optional: cloud browsers (no local Chrome needed)
# Free key at cloud.browser-use.com/new-api-key
BROWSER_USE_API_KEY=
# Set to auto-spawn cloud browser when local Chrome is unavailable
BU_AUTOSPAWN=1
2 changes: 0 additions & 2 deletions .github/CODEOWNERS

This file was deleted.

38 changes: 0 additions & 38 deletions .github/ISSUE_TEMPLATE/bug_report.md

This file was deleted.

20 changes: 0 additions & 20 deletions .github/ISSUE_TEMPLATE/feature_request.md

This file was deleted.

44 changes: 0 additions & 44 deletions .github/dependabot.yml

This file was deleted.

12 changes: 0 additions & 12 deletions .github/pull_request_template.md

This file was deleted.

141 changes: 0 additions & 141 deletions .github/workflows/ci.yml

This file was deleted.

Loading