MAINT: Update scorer evaluation metrics for v1.1.0 - #2569
Closed
varunj-msft wants to merge 1 commit into
Closed
Conversation
Regenerates scorer evaluation metrics with `python -m build_scripts.evaluate_scorers` for the v1.1.0 release. Adds 14 rows tagged `pyrit_version: 1.1.0` across 8 registry files (6 harm, objective, refusal). The files are append-only, so all historical rows are preserved. The numbers are consistent with prior releases: SelfAskRefusalScorer accuracy 0.971-0.990, SelfAskLikertScorer mean absolute error 0.169-0.374, and TrueFalseCompositeScorer stays high-precision/low-recall (0.979/0.241) well inside its historical band. No scorer regression. fairness_bias and sexual are not included. Their datasets label the harm category `bias` and `sexual_content`, while the harm definitions declare `fairness_bias` and `sexual`, so evaluation raises before writing a row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Superseded by #2571, which is opened from the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Regenerates the scorer evaluation metrics for the v1.1.0 release, per the "Before Publishing" checklist in
doc/contributing/10_release_process.md(python -m build_scripts.evaluate_scorers).Adds 14 rows tagged
pyrit_version: 1.1.0across 8 registry files. The registry files are append-only, so every historical row is preserved — the diff is+14 / -0.harm/{exploits,hate_speech,information_integrity,privacy,self_harm,violence}_metrics.jsonlSelfAskLikertScorerobjective/objective_achieved_metrics.jsonlTrueFalseCompositeScorer,TrueFalseInverterScorer×2refusal_scorer/refusal_metrics.jsonlSelfAskRefusalScorerValidation
The numbers were checked against every historical row for the same scorer, not just eyeballed:
SelfAskRefusalScorer×5 — accuracy 0.971–0.990, F1 0.979–0.993.SelfAskLikertScorer×6 — mean absolute error 0.169–0.374 on a 1–5 Likert scale.TrueFalseInverterScorer×2 — accuracy 0.884 / 0.896.TrueFalseCompositeScorer— accuracy 0.630, precision 0.979, recall 0.241. The low recall looks alarming in isolation, so it was compared against the full history for this scorer: recall has always been 0.241–0.439 with precision 0.940–0.988. This scorer is inherently high-precision/low-recall (ANDaggregation), and the new row sits inside that band — closest to the 0.12.1.dev0 run (accuracy 0.628, recall 0.246).No scorer regression detected in this release.
Not included:
fairness_biasandsexualThese two datasets did not produce rows. Their CSVs label the harm category
biasandsexual_content, whilepyrit/datasets/harm_definition/{fairness_bias,sexual}.yamldeclarefairness_biasandsexual.ScorerEvaluator._select_evaluation_scorecompares the two and raises before a row can be written:The other 6 of 8 harm datasets have matching names and evaluated normally. Tracked in #2570. Not addressed here, to keep this PR to data only.