FEAT: Add LayaRefusalScorer, a local CPU refusal scorer

Adds a true/false refusal scorer that runs on the machine running PyRIT,
with no API call and no per-response cost.

The scorer puts the refusal question to Laya (Convai Innovations, Apache 2.0),
an encoder that answers typed questions in one forward pass, takes the
representation Laya forms of each answer option rather than its own verdict,
and reads that with a logistic head trained on PyRIT's own human-labeled
refusal rows. The head is trained on first use from the pinned datasets, so
no opaque weights file ships and the whole model is reproducible from the
repository at a pinned dataset and checkpoint state.

Trained on refusal.csv and evaluated on refusal_extra.csv it is right on
96.9 percent of rows, and 91.4 percent the other way round. Excluding rows
whose response text also appears in training gives 97.5 percent of 40 rows
and 91.2 percent of 68 rows. SelfAskRefusalScorer with GPT-4o reaches 97 to
98 percent on the same rows. Laya's own verdict without the trained head
agrees with the labels on 53 to 71 percent of rows depending on the order
its answer options are presented in, so both orders are averaged and the
trained head is what makes it usable.

Scoring is two forward passes, about a second per response on CPU.

Author: WatchTree-19

doc/code/scoring/1_true_false_scorers.ipynb

@@ -508,7 +508,33 @@
     "Do not apply a second chat wrapper to the bundled prompt. If using a chat server that\n",
     "supplies its own formatting, pass a matching `prompt_template` explicitly.\n",
     "\n",
-    "All five need their respective endpoints/credentials even though they are not \"self-ask\"."
+    "All five need their respective endpoints/credentials even though they are not \"self-ask\".\n",
+    "\n",
+    "## Local classifier scorers\n",
+    "\n",
+    "- **`LayaRefusalScorer`**: detects refusals on the machine running PyRIT, with no API call.\n",
+    "  It puts the refusal question to [Laya](https://huggingface.co/convaiinnovations/laya), an\n",
+    "  Apache 2.0 encoder that answers typed questions in one forward pass, takes the representation\n",
+    "  Laya forms of each answer option, and reads it with a logistic head trained on PyRIT's own\n",
+    "  human-labeled refusal rows. Scoring is two forward passes, roughly a second per response on\n",
+    "  CPU, with no per-response cost.\n",
+    "\n",
+    "Trained on one of PyRIT's two refusal datasets and evaluated on the other, it is right on\n",
+    "96.9% and 91.4% of rows respectively (97.5% and 91.2% once rows whose response text also\n",
+    "appears in training are excluded). `SelfAskRefusalScorer` with GPT-4o reaches 97-98% on the\n",
+    "same rows. Laya's own verdict, used as it comes without the trained head, agrees with the\n",
+    "labels on 53-71% of rows, so the training step is what makes it usable.\n",
+    "\n",
+    "The head trains itself on first use from the pinned datasets, which takes a few minutes of CPU\n",
+    "once, and `abstain_band` returns an undetermined score for the uncertain tail so it can be\n",
+    "routed to an LLM judge:\n",
+    "\n",
+    "```python\n",
+    "from pyrit.score import LayaRefusalScorer\n",
+    "\n",
+    "scorer = LayaRefusalScorer()  # pip install laya\n",
+    "scores = await scorer.score_text_async(\"I'm sorry, I can't help with that.\")\n",
+    "```"
    ]
   },
   {

doc/code/scoring/1_true_false_scorers.py

@@ -290,6 +290,32 @@
 # supplies its own formatting, pass a matching `prompt_template` explicitly.
 #
 # All five need their respective endpoints/credentials even though they are not "self-ask".
+#
+# ## Local classifier scorers
+#
+# - **`LayaRefusalScorer`**: detects refusals on the machine running PyRIT, with no API call.
+#   It puts the refusal question to [Laya](https://huggingface.co/convaiinnovations/laya), an
+#   Apache 2.0 encoder that answers typed questions in one forward pass, takes the representation
+#   Laya forms of each answer option, and reads it with a logistic head trained on PyRIT's own
+#   human-labeled refusal rows. Scoring is two forward passes, roughly a second per response on
+#   CPU, with no per-response cost.
+#
+# Trained on one of PyRIT's two refusal datasets and evaluated on the other, it is right on
+# 96.9% and 91.4% of rows respectively (97.5% and 91.2% once rows whose response text also
+# appears in training are excluded). `SelfAskRefusalScorer` with GPT-4o reaches 97-98% on the
+# same rows. Laya's own verdict, used as it comes without the trained head, agrees with the
+# labels on 53-71% of rows, so the training step is what makes it usable.
+#
+# The head trains itself on first use from the pinned datasets, which takes a few minutes of CPU
+# once, and `abstain_band` returns an undetermined score for the uncertain tail so it can be
+# routed to an LLM judge:
+#
+# ```python
+# from pyrit.score import LayaRefusalScorer
+#
+# scorer = LayaRefusalScorer()  # pip install laya
+# scores = await scorer.score_text_async("I'm sorry, I can't help with that.")
+# ```
 # %% [markdown]
 # ## Multimodal scorers
 #

pyrit/score/__init__.py

@@ -78,6 +78,7 @@
     from pyrit.score.true_false.decoding_scorer import DecodingScorer
     from pyrit.score.true_false.float_scale_threshold_scorer import FloatScaleThresholdScorer
     from pyrit.score.true_false.gandalf_scorer import GandalfScorer
+    from pyrit.score.true_false.laya_refusal_scorer import LayaRefusalScorer
     from pyrit.score.true_false.llamaguard_parser import LLAMAGUARD_3_CATEGORY_CODES, parse_llamaguard_response
     from pyrit.score.true_false.llamaguard_policy import LlamaGuardCategory, LlamaGuardPolicy
     from pyrit.score.true_false.llamaguard_scorer import (
@@ -184,6 +185,7 @@
     "TraceAcquisitionError": "pyrit.score.observation.trace_client",
     "TraceClient": "pyrit.score.observation.trace_client",
     "JsonSchemaResponseHandler": "pyrit.score.response_handler",
+    "LayaRefusalScorer": "pyrit.score.true_false.laya_refusal_scorer",
     "LDAPInjectionOutputScorer": "pyrit.score.true_false.regex.ldap_injection_output_scorer",
     "LikertScaleEvalFiles": "pyrit.score.float_scale.self_ask_likert_scorer",
     "LikertScale": "pyrit.score.float_scale.likert_scale",