Skip to content

fix(metrics): score ChrF against each reference for multiple references - #7487

Open
chuenchen309 wants to merge 1 commit into
comet-ml:mainfrom
chuenchen309:fix/chrf-multi-reference-scoring
Open

fix(metrics): score ChrF against each reference for multiple references#7487
chuenchen309 wants to merge 1 commit into
comet-ml:mainfrom
chuenchen309:fix/chrf-multi-reference-scoring

Conversation

@chuenchen309

Copy link
Copy Markdown
Contributor

Details

ChrF.score accepts reference: Union[str, Sequence[str]] and builds a list of references, so multiple references are a documented, type-hinted feature. But the default NLTK backend passed that whole list straight to nltk.translate.chrf_score.sentence_chrf(references, candidate), whose reference argument is a single reference string — not a list. NLTK misinterprets the list and returns a wrong score: a candidate identical to one of the references scored ~0.26 instead of 1.0 (even lower than scoring against that reference alone).

This scores the candidate against each reference separately and takes the best match — standard multi-reference chrF semantics (as in sacrebleu / HuggingFace evaluate). Single-reference scoring is unchanged, and the custom chrf_fn injection contract is preserved (it still receives the full references list).

  • Before: ChrF().score(output="hello world", reference=["completely different text here", "hello world"]).value0.2552
  • After: → 1.0

Verified against the real installed code before/after. Added test_chrf_metric__multiple_references__scores_against_best_match (the existing ChrF tests only covered single-reference and the custom-chrf_fn path, so the real multi-reference NLTK path was untested) — red before the fix, green after. Full ChrF test set: 3 passed. ruff check + ruff format: clean.

Change checklist

  • User facing
  • Documentation update

Issues

  • Resolves #

AI-WATERMARK

AI-WATERMARK: yes

  • Tools: Claude Code
  • Model(s): Claude (Anthropic)
  • Scope: Bug found via an AI-assisted review pass over sdks/python/src/opik/evaluation/metrics/; the fix and regression test were AI-drafted and then independently reproduced and verified (red-before/green-after against the real installed code, ruff clean) before submitting.

ChrF.score accepts `reference: Union[str, Sequence[str]]` and builds a list
of references, i.e. multiple references are a documented, type-hinted
feature. But the default NLTK backend passed that whole list straight to
`nltk.translate.chrf_score.sentence_chrf(references, candidate)`, whose
`reference` argument is a single reference string, not a list. NLTK then
misinterprets the list and returns a wrong score: a candidate identical to
one of the references scored ~0.26 instead of 1.0 — even lower than scoring
against that reference alone.

Score the candidate against each reference separately and take the best
match, which is the standard multi-reference chrF semantics (as in
sacrebleu / HuggingFace evaluate). Single-reference scoring is unchanged,
and the custom `chrf_fn` injection contract is preserved (it still receives
the full references list and decides how to combine them).

Verified against the real installed code before/after: a candidate equal to
`refs[1]` returned 0.2552 before and 1.0 after; a candidate matching either
of two references now returns 1.0; a no-match case scores low as expected.
Added `test_chrf_metric__multiple_references__scores_against_best_match`
(the existing ChrF tests only exercised single-reference and a custom-fn
path, so the real multi-ref NLTK path was untested) — red before the fix,
green after, confirmed by stashing the source change. Full ChrF test set:
3 passed. ruff check + ruff format: clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@chuenchen309
chuenchen309 requested a review from a team as a code owner July 16, 2026 01:55
@github-actions github-actions Bot added 🟢 size/S python Pull requests that update Python code tests Including test files, or tests related like configuration. Python SDK and removed 🟢 size/S labels Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Python SDK python Pull requests that update Python code tests Including test files, or tests related like configuration.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant