[SDK] fix(evaluation): don't mutate the caller's scoring_metrics list

[chuenchen309]

Details

_wrap_scoring_functions (evaluator.py:1726) appends the wrapped scorer functions into the list the caller passed in:

if scoring_metrics:
    scoring_metrics.extend(function_metrics)   # mutates the caller's list
else:
    scoring_metrics = function_metrics          # rebinds — safe

When a user passes both scoring_metrics and scoring_functions and reuses the metrics list across calls — the natural way to write it — each call appends another wrapper. Run N executes every scorer function N times:

scoring_metrics = [Equals()]

for _ in range(3):
    evaluation.evaluate_on_dict_items(
        items=items, task=task,
        scoring_metrics=scoring_metrics,
        scoring_functions=[my_scorer],
    )

call 1: metrics used = ['equals_metric', 'my_scorer']
call 2: metrics used = ['equals_metric', 'my_scorer', 'my_scorer']
call 3: metrics used = ['equals_metric', 'my_scorer', 'my_scorer', 'my_scorer']
user's list AFTER   = ['equals_metric', 'my_scorer', 'my_scorer', 'my_scorer']

That means N duplicate ScoreResults per item, N duplicate feedback scores written to the backend, N× the LLM cost for an LLM-backed scorer function, and a skewed aggregate_evaluation_scores() (the values list is padded with duplicates, so the count and std are wrong). The user's own list is left mutated afterwards too.

The else branch already rebinds instead of mutating, so this just makes the if branch consistent with it.

Reachable from every public entry point that takes both arguments: evaluate(), evaluate_experiment(), evaluate_prompt(), evaluate_optimization_trial(), evaluate_resume(), evaluate_on_dict_items().

Change checklist

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Issues

No existing issue — happy to open one if the team prefers that order for small SDK fixes.

Testing

Added test_evaluate_on_dict_items__scoring_metrics_list_is_not_mutated, which reuses one scoring_metrics list across two evaluate_on_dict_items calls and asserts both that the list is untouched and that the second run scores each metric once.

Verified it fails before the fix (assert ['equals_metric', 'custom_scorer', 'custom_scorer'] == ['equals_metric']) and passes after.

Existing scoring_functions coverage at test_evaluate.py:2905 only passes scoring_functions without a concurrent non-empty scoring_metrics, so it takes the safe else branch and never exercised this path.

tests/unit/evaluation/ — 744 passed, matching the pre-change baseline of 743 plus the new test. The 48 failures in that directory are identical before and after (diffed the failure lists item by item); they're pre-existing litellm-related environment failures, unrelated to this change. ruff check and ruff format --check clean.

Documentation

No documentation changes needed — this restores the documented behaviour rather than changing it.

AI-WATERMARK: yes
Tools: Claude Code
Model(s): Claude Opus 4.8
Scope: Bug located with AI assistance and the fix drafted with it. I reproduced the duplication myself with a standalone script before writing anything, traced the mechanism to evaluator.py:1726 by reading the source, confirmed no open PR touches this function, and ran the tests and baseline comparison above myself.

[baz-reviewer]

Missing score-value assertions

evaluate_on_dict_items() is only checked for result names and scoring_metrics immutability, so regressions in metrics.Equals() or custom_scorer() would still pass as long as the shape matches — should we assert the ScoreResult.value fields too, e.g. both 1.0 for the matching output/expected_output case?

Want Baz to fix this for you? Activate Fixer

Other fix methods

Prompt for AI Agents

Before applying, verify this suggestion against the current code. In
sdks/python/tests/unit/evaluation/test_evaluate.py around lines 2964-2993 in
`test_evaluate_on_dict_items__scoring_metrics_list_is_not_mutated`, extend the
assertions after the `evaluation.evaluate_on_dict_items(...)` call to also verify the
`ScoreResult.value` fields. Specifically, assert that the equals metric score value is
`1.0` for the matching `output`/`expected_output` and that the `custom_scorer` score
value is also `1.0` (or the exact value your scorer returns). Implement the assertions
by locating score results by `score.name` (or otherwise ensuring the correct pairing) so
the test validates scoring correctness rather than only result shape/name ordering.

[petrotiurin]

Thank you for spotting this bug and for fixing it @chuenchen309 !