Repository navigation
Conversation
ToolCallStepSequenceMatchGrader's strict step-by-step mode scored parameters as an F1 over a flat set of argument names and values, so a call with swapped values (origin/destination) scored 1.0, a value matched regardless of which argument it belonged to, and the same nested object with a different key order scored 0.5. Build the set from (name, value) pairs and render nested dict/list values with json.dumps(sort_keys=True).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenJudge Version
0.2.5 (
mainat 15d22f2)Description
In strict step-by-step mode (
strict_mode=True, use_jaccard_similarity=False),ToolCallStepSequenceMatchGraderscores parameters with an F1 overcreate_tool_elements(), which puts every argument name and every argument value into one flat set. Values lose the argument they belong to, so:A swapped or misassigned argument gets a perfect score, and an equivalent nested object is penalised. A wrong value also always earns partial credit for its key (
{"q": "A"}vs{"q": "B"}scores 0.5).Change: build the set from
(name, value)pairs, rendering nested dict/list values withjson.dumps(sort_keys=True). Scalars are still compared viastr(), so1and"1"match as before. This is consistent with how #239 compares arguments in strict Jaccard mode (object key order ignored, array order significant).Expected-value updates. Three existing cases relied on the key-only credit:
test_strict_steps_match_repeated_tools_once:["A","B"]vs["A","A"](both directions) goes from 0.75 to 0.5. The unmatchedsearch(query=B)againstsearch(query=A)now scores 0 instead of 0.5.test_strict_steps_keep_different_tool_names_separategoes from 0.5 to 0.0. Every call there has the wrong value for its tool, and the test still checks that matches don't cross tool names (crossing would give 1.0).test_strict_steps_maximize_total_parameter_similarityis unchanged (5/6).Tests: added
test_strict_steps_compare_parameters_as_key_value_pairs(swapped strings, swapped ints, value placed under the wrong key, one of two wrong, nested key order, array order). Onmain9 cases fail. With this changetests/graders/agent/passes (250 passed, 49 skipped).Checklist
pre-commit run --all-filescommand (ran on the changed files: all hooks pass)