Skip to content

fix(agent): compare strict step parameters as key-value pairs - #249

Open
fxia0018 wants to merge 1 commit into
agentscope-ai:mainfrom
fxia0018:fix/strict-step-param-pairs
Open

fxia0018 wants to merge 1 commit into
agentscope-ai:mainfrom
fxia0018:fix/strict-step-param-pairs

Conversation

@fxia0018

Copy link
Copy Markdown
Contributor

OpenJudge Version

0.2.5 (main at 15d22f2)

Description

In strict step-by-step mode (strict_mode=True, use_jaccard_similarity=False), ToolCallStepSequenceMatchGrader scores parameters with an F1 over create_tool_elements(), which puts every argument name and every argument value into one flat set. Values lose the argument they belong to, so:

reference = {"origin": "北京", "destination": "上海"}
predicted = {"origin": "上海", "destination": "北京"}   # swapped
# elements: {"origin", "destination", "北京", "上海"} on both sides -> score 1.0

reference = {"a": 1, "b": 2}
predicted = {"a": 2, "b": 1}                             # -> score 1.0

reference = {"filter": {"lang": "en", "year": 2024}}
predicted = {"filter": {"year": 2024, "lang": "en"}}     # same object
# str() keeps insertion order -> values differ -> score 0.5

A swapped or misassigned argument gets a perfect score, and an equivalent nested object is penalised. A wrong value also always earns partial credit for its key ({"q": "A"} vs {"q": "B"} scores 0.5).

Change: build the set from (name, value) pairs, rendering nested dict/list values with json.dumps(sort_keys=True). Scalars are still compared via str(), so 1 and "1" match as before. This is consistent with how #239 compares arguments in strict Jaccard mode (object key order ignored, array order significant).

Expected-value updates. Three existing cases relied on the key-only credit:

  • test_strict_steps_match_repeated_tools_once: ["A","B"] vs ["A","A"] (both directions) goes from 0.75 to 0.5. The unmatched search(query=B) against search(query=A) now scores 0 instead of 0.5.
  • test_strict_steps_keep_different_tool_names_separate goes from 0.5 to 0.0. Every call there has the wrong value for its tool, and the test still checks that matches don't cross tool names (crossing would give 1.0).

test_strict_steps_maximize_total_parameter_similarity is unchanged (5/6).

Tests: added test_strict_steps_compare_parameters_as_key_value_pairs (swapped strings, swapped ints, value placed under the wrong key, one of two wrong, nested key order, array order). On main 9 cases fail. With this change tests/graders/agent/ passes (250 passed, 49 skipped).

Checklist

  • Code has been formatted with pre-commit run --all-files command (ran on the changed files: all hooks pass)
  • All tests are passing
  • Docstrings are in Google style
  • Related documentation has been updated (e.g. links, examples, etc.)
  • Code is ready for review

ToolCallStepSequenceMatchGrader's strict step-by-step mode scored
parameters as an F1 over a flat set of argument names and values, so a
call with swapped values (origin/destination) scored 1.0, a value matched
regardless of which argument it belonged to, and the same nested object
with a different key order scored 0.5.

Build the set from (name, value) pairs and render nested dict/list values
with json.dumps(sort_keys=True).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant