fix(metrics): score ChrF against each reference for multiple references - #7487
Open
chuenchen309 wants to merge 1 commit into
Open
fix(metrics): score ChrF against each reference for multiple references#7487chuenchen309 wants to merge 1 commit into
chuenchen309 wants to merge 1 commit into
Conversation
ChrF.score accepts `reference: Union[str, Sequence[str]]` and builds a list of references, i.e. multiple references are a documented, type-hinted feature. But the default NLTK backend passed that whole list straight to `nltk.translate.chrf_score.sentence_chrf(references, candidate)`, whose `reference` argument is a single reference string, not a list. NLTK then misinterprets the list and returns a wrong score: a candidate identical to one of the references scored ~0.26 instead of 1.0 — even lower than scoring against that reference alone. Score the candidate against each reference separately and take the best match, which is the standard multi-reference chrF semantics (as in sacrebleu / HuggingFace evaluate). Single-reference scoring is unchanged, and the custom `chrf_fn` injection contract is preserved (it still receives the full references list and decides how to combine them). Verified against the real installed code before/after: a candidate equal to `refs[1]` returned 0.2552 before and 1.0 after; a candidate matching either of two references now returns 1.0; a no-match case scores low as expected. Added `test_chrf_metric__multiple_references__scores_against_best_match` (the existing ChrF tests only exercised single-reference and a custom-fn path, so the real multi-ref NLTK path was untested) — red before the fix, green after, confirmed by stashing the source change. Full ChrF test set: 3 passed. ruff check + ruff format: clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details
ChrF.scoreacceptsreference: Union[str, Sequence[str]]and builds a list of references, so multiple references are a documented, type-hinted feature. But the default NLTK backend passed that whole list straight tonltk.translate.chrf_score.sentence_chrf(references, candidate), whosereferenceargument is a single reference string — not a list. NLTK misinterprets the list and returns a wrong score: a candidate identical to one of the references scored ~0.26 instead of 1.0 (even lower than scoring against that reference alone).This scores the candidate against each reference separately and takes the best match — standard multi-reference chrF semantics (as in sacrebleu / HuggingFace
evaluate). Single-reference scoring is unchanged, and the customchrf_fninjection contract is preserved (it still receives the full references list).ChrF().score(output="hello world", reference=["completely different text here", "hello world"]).value→0.25521.0Verified against the real installed code before/after. Added
test_chrf_metric__multiple_references__scores_against_best_match(the existing ChrF tests only covered single-reference and the custom-chrf_fnpath, so the real multi-reference NLTK path was untested) — red before the fix, green after. Full ChrF test set: 3 passed.ruff check+ruff format: clean.Change checklist
Issues
AI-WATERMARK
AI-WATERMARK: yes
sdks/python/src/opik/evaluation/metrics/; the fix and regression test were AI-drafted and then independently reproduced and verified (red-before/green-after against the real installed code, ruff clean) before submitting.