feat(datasets): export Aegis violence rows as HumanLabeledDataset

[Ray Winkelman (rwinkelman)]

Summary

• Adds fetch_human_labeled_harm_dataset_async() on the Aegis 2.0 remote loader.
• Retains assistant response + response_label and maps them into HarmHumanLabeledEntry rows for scorer evaluation.
• Starts with the Violence category (CC-BY-4.0, clean mapping to violence.yaml).

Addresses #2475.

Test plan

• pytest tests/unit/datasets/test_aegis_human_labeled_dataset.py

[Ray Winkelman (rwinkelman)]

Extended with hate_speech profile: HATE_IDENTITY_HATE now auto-selects hate_speech.yaml and builds aegis_hate_speech HumanLabeledDataset.

[Ray Winkelman (rwinkelman)]

Follow-up: added malware→cyber.yaml, sexual, and suicide/self-harm harm_definition profiles + malware unit test.

[WatchTree-19]

Thanks for building this Ray Winkelman (@rwinkelman), and sorry for the slow response. Answering your encoding question, plus three things from reading the diff.

On binary safe/unsafe: yes, and there's evidence for it

Binary is the right call, and not just as a simplification. From the three-rater files in scorer_evals/harm/, agreement between annotators on the exact severity value is 42% in hate_speech_multi_score and 32% in violence_multi_score. Binarised at 0.5 those go to 82% and 69%. Almost all the human disagreement is in the severity number, not the harmful/not-harmful call.

So a graded encoding of an Aegis binary label would be inventing precision that your own gold set doesn't have between raters. {0.0, 1.0} is honest about what the source actually says.

One consequence worth documenting rather than fixing: any metric that treats human_scores as continuous will behave differently on an Aegis-derived set than on the native scorer_evals ones, because the distribution is degenerate at the endpoints. Not an argument against binary, just something a caller comparing the two shouldn't be surprised by.

The one I'd push back on: response_label_source is dropped

This is the substantive one.

Aegis carries response_label_source per row, and it takes the value human or llm_jury — your own test fixtures have both, e.g. row "id": "7" is "response_label_source": "llm_jury" and it lands in the dataset with human_scores == [1.0]. test_fetch_human_labeled_harm_dataset_violence asserts len(dataset.entries) == 2, and one of those two is jury-labelled.

So as written, LLM-jury output flows into a field called human_scores, on a class called HumanLabeledDataset, whose purpose is to measure LLM scorers.

That matters beyond naming. scorer_evals is the only clean instrument in the repo for asking whether a judge is any good. If jury labels enter it, some fraction of the benchmark becomes model output grading model output, and the number quietly stops meaning what it says. It's the same failure mode as the rest of this thread: an evaluation that returns a confident number rather than admitting its inputs are compromised.

Suggestion: a response_label_source filter defaulting to human-only, so the safe behaviour is the default and taking jury rows is an explicit opt-in.

label_sources: tuple[str, ...] = ("human",),

and skip rows whose response_label_source isn't in it. Callers who genuinely want the larger jury-labelled pool for training can pass ("human", "llm_jury") and will know what they've asked for. Worth logging the counts of each so the split is visible.

I don't know offhand what the human/jury ratio is in Aegis 2.0, so this could turn out to cost most of the rows. If it does, that's an argument for surfacing the split loudly, not for keeping them mixed.

The prompt is dropped from the conversation

Each entry is built with a single MessagePiece(role="assistant", ...) and the user prompt isn't carried. For harm scoring that's a real loss: the same response can be harmful or benign depending on what was asked, and PyRIT's own scorers see the exchange. A refusal to a violent request and the same words offered unprompted score differently.

Since Aegis gives you prompt right there, including it as a preceding role="user" piece in the same conversation looks close to free. Unless there's a reason to keep these single-turn that I'm missing.

Small one: validate the harm definition before the loop

HarmDefinition.from_yaml(harm_definition) runs after the full iteration over every split. For an unmapped category, _resolve_human_labeled_harm_profile falls back to f"{pyrit_name}.yaml", so the failure is a bad filename that only raises once the whole dataset has been walked. Moving that call above the loop makes it fail immediately.

Same area: if not response_value or not response_label: continue skips silently. A count in the log would make it obvious if a schema change starts dropping rows.

---

None of this is an argument against the shape, which I think is right, and the _HUMAN_LABELED_HARM_PROFILES mapping is cleaner than what I had in mind when I filed the issue. The response_label_source point is the only one I'd consider blocking, and only because scorer_evals is hard to un-contaminate once it is.