Skip to content

Add CrossSource to 5c · RAG / retrieval evaluation - #83

Open
zoeb-nomi wants to merge 1 commit into
benchflow-ai:mainfrom
zoeb-nomi:patch-1
Open

Add CrossSource to 5c · RAG / retrieval evaluation#83
zoeb-nomi wants to merge 1 commit into
benchflow-ai:mainfrom
zoeb-nomi:patch-1

Conversation

@zoeb-nomi

Copy link
Copy Markdown

Adds CrossSource to 5c · RAG / retrieval evaluation — an open evaluation harness measuring citation accuracy in RAG over public court opinions.

Why it clears the bar (real data + code, not a generic "you need evals" take):

  • Validated LLM-as-judge: 15/15 blind human–judge agreement on a stratified sample.
  • The validation pass itself caught a real harness bug the judge couldn't see — consecutive citations producing punctuation-only claim spans scored as citation failures. Fixing it moved citation precision 0.981 → 0.994 under strict citation discipline.
  • Claim-level citation precision & recall, plus a failure taxonomy that separates prompting failures from retrieval failures.

Repo (canonical, URL verified): https://github.com/zoeb-nomi/crosssource

One addition, placed in 5c immediately after Tonic Validate; entry annotated per CONTRIBUTING.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant