Skip to content

MAINT: Update scorer evaluation metrics for v1.1.0 - #2569

Closed
varunj-msft wants to merge 1 commit into
mainfrom
copilot/scorer-eval-metrics-v1.1.0
Closed

MAINT: Update scorer evaluation metrics for v1.1.0#2569
varunj-msft wants to merge 1 commit into
mainfrom
copilot/scorer-eval-metrics-v1.1.0

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Regenerates the scorer evaluation metrics for the v1.1.0 release, per the "Before Publishing" checklist in doc/contributing/10_release_process.md (python -m build_scripts.evaluate_scorers).

Adds 14 rows tagged pyrit_version: 1.1.0 across 8 registry files. The registry files are append-only, so every historical row is preserved — the diff is +14 / -0.

File Rows Scorer
harm/{exploits,hate_speech,information_integrity,privacy,self_harm,violence}_metrics.jsonl 1 each SelfAskLikertScorer
objective/objective_achieved_metrics.jsonl 3 TrueFalseCompositeScorer, TrueFalseInverterScorer ×2
refusal_scorer/refusal_metrics.jsonl 5 SelfAskRefusalScorer

Validation

The numbers were checked against every historical row for the same scorer, not just eyeballed:

  • SelfAskRefusalScorer ×5 — accuracy 0.971–0.990, F1 0.979–0.993.
  • SelfAskLikertScorer ×6 — mean absolute error 0.169–0.374 on a 1–5 Likert scale.
  • TrueFalseInverterScorer ×2 — accuracy 0.884 / 0.896.
  • TrueFalseCompositeScorer — accuracy 0.630, precision 0.979, recall 0.241. The low recall looks alarming in isolation, so it was compared against the full history for this scorer: recall has always been 0.241–0.439 with precision 0.940–0.988. This scorer is inherently high-precision/low-recall (AND aggregation), and the new row sits inside that band — closest to the 0.12.1.dev0 run (accuracy 0.628, recall 0.246).

No scorer regression detected in this release.

Not included: fairness_bias and sexual

These two datasets did not produce rows. Their CSVs label the harm category bias and sexual_content, while pyrit/datasets/harm_definition/{fairness_bias,sexual}.yaml declare fairness_bias and sexual. ScorerEvaluator._select_evaluation_score compares the two and raises before a row can be written:

ValueError: Scorer evaluation requires a score for harm category 'bias',
but the single score returned is categorized as ['fairness_bias'].

The other 6 of 8 harm datasets have matching names and evaluated normally. Tracked in #2570. Not addressed here, to keep this PR to data only.

Regenerates scorer evaluation metrics with `python -m build_scripts.evaluate_scorers`
for the v1.1.0 release. Adds 14 rows tagged `pyrit_version: 1.1.0` across 8 registry
files (6 harm, objective, refusal). The files are append-only, so all historical rows
are preserved.

The numbers are consistent with prior releases: SelfAskRefusalScorer accuracy
0.971-0.990, SelfAskLikertScorer mean absolute error 0.169-0.374, and
TrueFalseCompositeScorer stays high-precision/low-recall (0.979/0.241) well inside its
historical band. No scorer regression.

fairness_bias and sexual are not included. Their datasets label the harm category
`bias` and `sexual_content`, while the harm definitions declare `fairness_bias` and
`sexual`, so evaluation raises before writing a row.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@varunj-msft

Copy link
Copy Markdown
Contributor Author

Superseded by #2571, which is opened from the varunj-msft/PyRIT fork to match the convention used by every other release PR this cycle. Same commit, same +14/−0 diff. Closing this one and deleting the upstream branch.

@varunj-msft varunj-msft closed this Sep 4, 2026
@varunj-msft
varunj-msft deleted the copilot/scorer-eval-metrics-v1.1.0 branch September 4, 2026 02:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant