Repository navigation
FEAT: add MessageToolCallScorer for tool calls recorded in stored messages - #2854
Merged
Richard Lundeen (richlundeen) merged 4 commits intoOct 4, 2026
Conversation
…sages Scores the ToolsCalled condition from function_call and function_call_output pieces, with no trace pipeline. A tool counts when a model-authored call is paired by call ID with a later output in the conversation through the scored response. PyRIT tolerant-mode dispatch errors do not count. Stored messages are partial evidence, so the scorer returns true or undetermined, never false. Handles both the Responses and Chat Completions call formats. No observation payload or replay yet; that needs a message-scoped tool-event payload.
WatchTree-19
force-pushed
the
feat/message-tool-call-scorer
branch
from
October 1, 2026 16:21
81f5877 to
c3fd286
Compare
Contributor
Author
|
Rebased onto main after #2852 and #2832. The scorer now reads calls and outputs through the shared FunctionCallContent and FunctionOutputContent models instead of its own JSON parsing, uses the async memory API, and ignores simulated_tool outputs as well as simulated_assistant calls, with a new test for that case. 31 tests in test_message_tool_call_scorer.py pass, and tests/unit/score is green (2926 passed). |
Separate dispatch status from tool-returned data and guard legacy structured errors. Add metadata serialization, regression coverage, and documentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
approved these changes
Oct 1, 2026
Update the scorer test helper to use the async memory API so deprecation warnings do not fail CI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
enabled auto-merge
October 2, 2026 04:44
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 2, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
MessageToolCallScorer, the message-side counterpart toOtelToolCallScorer. It scores the sameToolsCalledcondition from the function-call pieces PyRIT already stores, so it works without a trace pipeline.A tool counts when a model-authored
function_callpiece is paired by call ID with a laterfunction_call_outputpiece in the scored message's conversation, up to and including that message, which is the same scoperesolve_message_trace_scopeuses. Both the Responses (call_id,name) and Chat Completions (id,function.name) formats are read. Requested calls with no output do not count, calls insimulated_assistanthistory are ignored, and PyRIT's tolerant-mode dispatch outputs (function_not_found,missing_function_name,malformed_arguments) do not count, since the function never ran. An error returned by the tool itself still counts, in line withToolsCalledcounting an attempt regardless of success.Stored messages are partial evidence: hosted tools and several response section types are not persisted, and targets that run tools themselves store no outputs. The scorer therefore returns true when every required tool ran and undetermined otherwise, never false, and it composes with
OtelToolCallScorerunderTrueFalseScoreAggregator.OR. It does not create observations, soscore_observation_asyncis not supported for it.Tests and Documentation
tests/unit/score/test_message_tool_call_scorer.py(30 tests): both call formats, unpaired requests, dispatch errors versus tool-reported errors, call ID pairing, output ordering, later turns and other conversations, role and data type filtering, multiple required tools, an OR composite withOtelToolCallScorer, and an end-to-endPromptSendingAttackthrough a mockedOpenAIResponseTargettool loop.tests/unit/score,tests/unit/docsandtests/unit/registrypass; ruff and ty are clean.doc/code/scoring/5_tool_call_scorer.pyand synced the.ipynbwith JupyText. No code cells changed, so no re-execution was needed.