MAINT: Update scorer evaluation metrics for v1.1.0

[varunj-msft]

Summary

Regenerates the scorer evaluation metrics for the v1.1.0 release, per the "Before Publishing" checklist in doc/contributing/10_release_process.md (python -m build_scripts.evaluate_scorers).

Data only — 8 registry files, +20 / −0, zero code. The registry files are append-only, so every historical row is preserved.

Of the 29 registered scorers: 20 ran and wrote a row, 7 were skipped because the registry already held a sufficient matching entry (RegistryUpdateBehavior.SKIP_IF_EXISTS skips when the dataset/harm-definition versions match and the existing num_scorer_trials is at least the requested 3), and 2 raised the error described below.

Note on the 7 skipped: 3 are threshold wrappers, but 4 are AzureContentFilterScorer harm baselines that do have datasets. Those 4 would hit the same category mismatch as #2570 if they were re-evaluated — the skip short-circuits before evaluation, so the bug is latent rather than fixed. See #2570.

File	Rows	Scorer
harm/{exploits,hate_speech,information_integrity,privacy,self_harm,violence}_metrics.jsonl	1 each	SelfAskLikertScorer
objective/objective_achieved_metrics.jsonl	7	TrueFalseCompositeScorer ×2, TrueFalseInverterScorer ×2, FloatScaleThresholdScorer, SelfAskTrueFalseScorer ×2
refusal_scorer/refusal_metrics.jsonl	7	SelfAskRefusalScorer

All 20 rows are tagged pyrit_version: 1.1.0.

Validation

Verified programmatically rather than by eye — every row is a pure append (all pre-existing bytes unchanged), parses as valid JSON, and is unique by (class_name, hash, eval_hash) so no scorer was written twice.

Metric values were compared against every historical row for the same scorer class:

• SelfAskRefusalScorer ×7 — accuracy 0.971–0.990, F1 0.979–0.993 (historical 0.962–1.000).
• SelfAskLikertScorer ×6 — mean absolute error 0.169–0.374 on a 1–5 Likert scale.
• TrueFalseInverterScorer ×2 — accuracy 0.884 / 0.896 (historical 0.785–0.894).
• FloatScaleThresholdScorer — accuracy 0.747, recall 0.487 (historical 0.585–0.757 / 0.152–0.503).
• SelfAskTrueFalseScorer ×2 — accuracy 0.625 / 0.729 (historical 0.628–0.800). These run at temperature 0.9, and 0.625 is within one standard error (~0.024 on 395 samples) of the historical floor.
• TrueFalseCompositeScorer ×2 — accuracy 0.630 / 0.742, precision 0.979 / 0.989, recall 0.241 / 0.471. The low recall looks alarming in isolation, so it was checked against the full history for this scorer: recall has always been 0.241–0.439 with precision 0.940–0.988. It is inherently high-precision/low-recall (AND aggregation).

No scorer regression detected in this release.

Not included: likert_fairness_bias_gpt4o and likert_sexual_gpt4o

Their datasets label the harm category bias and sexual_content, while pyrit/datasets/harm_definition/{fairness_bias,sexual}.yaml declare fairness_bias and sexual. ScorerEvaluator._select_evaluation_score compares the two and raises before a row can be written:

ValueError: Scorer evaluation requires a score for harm category 'bias',
but the single score returned is categorized as ['fairness_bias'].

This is a namespace collision rather than a typo — sexual_content and bias are real names in PyRIT's harm taxonomy, while fairness_bias and sexual are Likert scale identifiers. Tracked in #2570 with a suggested fix; deliberately not addressed here so this PR stays data-only. These two files keep their 0.14.0.dev0 metrics.

Note on the release branch

These metrics files ship inside the wheel (MANIFEST.in: recursive-include pyrit *.jsonl). Cherry-picking this into releases/v1.1.0 would change the built artifact and invalidate the validation already completed at d9dae48ce. Recommend main only.

[varunj-msft]

Evaluation run is complete: ok=6 failed=0 on the resumed scorers, 27 of 29 scorers evaluated, 20 rows.

Because the original run was killed mid-flight by a container restart (at 23/29) and these registry files are append-only, the two risks were double-written rows and a torn final line. Both were checked explicitly against the pushed commit 37cbef211, not just my working tree:

• Resumed, not restarted. I reconstructed the ordered scorer list from the registry and re-ran only indices 24–29, so scorers 1–23 were never re-executed. Re-running the whole script would have appended a second copy of all 23.
• No duplicates — every row across all 10 registry files is unique by (class_name, hash, eval_hash, pyrit_version).
• Pure append — for all 10 registries the pre-existing rows are byte-identical to main; nothing was rewritten.
• No torn writes — every line in every registry parses as JSON, and each file ends with exactly one newline.
• Nothing stray — of the 38 tracked scorer_evals files, exactly the intended 8 changed.
• Semantically consistent — every new row has num_scorer_trials = 3, all rates within [0,1], and num_responses equals the real dataset size (80 / 59 / 76 / 93 / 93 / 96 for the harm sets, 105 refusal, 395 objective across its 14 CSVs).
• Metrics inside historical bands for every scorer class, so no quality regression.

The 8 files on the branch were also confirmed byte-identical to the locally verified ones.

Only likert_fairness_bias_gpt4o and likert_sexual_gpt4o are missing, blocked by the namespace collision in #2570. Data-only change; no pyrit/** code touched.