Skip to content

FEAT: add MessageToolCallScorer for tool calls recorded in stored messages - #2854

Merged
Richard Lundeen (richlundeen) merged 4 commits into
microsoft:mainfrom
WatchTree-19:feat/message-tool-call-scorer
Oct 4, 2026
Merged

Richard Lundeen (richlundeen) merged 4 commits into
microsoft:mainfrom
WatchTree-19:feat/message-tool-call-scorer

Conversation

@WatchTree-19

Copy link
Copy Markdown
Contributor

Description

Adds MessageToolCallScorer, the message-side counterpart to OtelToolCallScorer. It scores the same ToolsCalled condition from the function-call pieces PyRIT already stores, so it works without a trace pipeline.

A tool counts when a model-authored function_call piece is paired by call ID with a later function_call_output piece in the scored message's conversation, up to and including that message, which is the same scope resolve_message_trace_scope uses. Both the Responses (call_id, name) and Chat Completions (id, function.name) formats are read. Requested calls with no output do not count, calls in simulated_assistant history are ignored, and PyRIT's tolerant-mode dispatch outputs (function_not_found, missing_function_name, malformed_arguments) do not count, since the function never ran. An error returned by the tool itself still counts, in line with ToolsCalled counting an attempt regardless of success.

Stored messages are partial evidence: hosted tools and several response section types are not persisted, and targets that run tools themselves store no outputs. The scorer therefore returns true when every required tool ran and undetermined otherwise, never false, and it composes with OtelToolCallScorer under TrueFalseScoreAggregator.OR. It does not create observations, so score_observation_async is not supported for it.

Tests and Documentation

  • tests/unit/score/test_message_tool_call_scorer.py (30 tests): both call formats, unpaired requests, dispatch errors versus tool-reported errors, call ID pairing, output ordering, later turns and other conversations, role and data type filtering, multiple required tools, an OR composite with OtelToolCallScorer, and an end-to-end PromptSendingAttack through a mocked OpenAIResponseTarget tool loop.
  • Each check in the scorer was mutation-tested: all 11 deliberate breaks fail at least one test.
  • tests/unit/score, tests/unit/docs and tests/unit/registry pass; ruff and ty are clean.
  • Added a markdown section to doc/code/scoring/5_tool_call_scorer.py and synced the .ipynb with JupyText. No code cells changed, so no re-execution was needed.

…sages

Scores the ToolsCalled condition from function_call and function_call_output
pieces, with no trace pipeline. A tool counts when a model-authored call is
paired by call ID with a later output in the conversation through the scored
response. PyRIT tolerant-mode dispatch errors do not count. Stored messages
are partial evidence, so the scorer returns true or undetermined, never false.

Handles both the Responses and Chat Completions call formats. No observation
payload or replay yet; that needs a message-scoped tool-event payload.
@WatchTree-19
WatchTree-19 force-pushed the feat/message-tool-call-scorer branch from 81f5877 to c3fd286 Compare October 1, 2026 16:21
@WatchTree-19

Copy link
Copy Markdown
Contributor Author

Rebased onto main after #2852 and #2832. The scorer now reads calls and outputs through the shared FunctionCallContent and FunctionOutputContent models instead of its own JSON parsing, uses the async memory API, and ignores simulated_tool outputs as well as simulated_assistant calls, with a new test for that case. 31 tests in test_message_tool_call_scorer.py pass, and tests/unit/score is green (2926 passed).

Separate dispatch status from tool-returned data and guard legacy structured errors. Add metadata serialization, regression coverage, and documentation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Update the scorer test helper to use the async memory API so deprecation warnings do not fail CI.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 2, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 2, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 4, 2026
Merged via the queue into microsoft:main with commit 2596f34 Oct 4, 2026
52 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants