Skip to content

Feat/garak exploitation scenario - #2743

Closed
Joel (aspire488) wants to merge 14 commits into
microsoft:mainfrom
aspire488:feat/garak-exploitation-scenario
Closed

Joel (aspire488) wants to merge 14 commits into
microsoft:mainfrom
aspire488:feat/garak-exploitation-scenario

Conversation

@aspire488

Copy link
Copy Markdown

Description

Tests and Documentation

Summary

Adds a PyRIT-native implementation of Garak's exploitation probe family.

What's included

  • "Exploitation" scenario with:
    • "JinjaTemplatePythonInjection"
    • "SQLInjectionEcho"
  • Curated Garak-derived payload datasets and reusable echo template.
  • Payload provenance and expected-output metadata.
  • Payload-specific "SubStringScorer" instances.
  • Stable payload-hash-based atomic attack names.
  • Bounded payload execution through "max_payloads_per_technique".
  • Generic baseline disabled because exploitation success is payload-specific.
  • Focused unit tests covering:
    • Technique expansion
    • Seed-group construction
    • Payload metadata
    • Scorer selection
    • Stable attack naming
  • Updated scanner documentation and notebook examples.

Safety / scope

The implementation tests whether a target model emits exploit material; it does not execute the generated Python or SQL against a runtime, database, or external system.

The initial datasets use a small non-destructive subset of the upstream Garak payload families.

Reference

Inspired by Garak's "exploitation" probe:

https://github.com/NVIDIA/garak/blob/main/garak/probes/exploitation.py

Closes #2534.

@romanlutz

Copy link
Copy Markdown
Contributor

Seems like a copy of #2576 which is covering the exploitation scenario. I'll close unless there's a reason for having this additionally.

@aspire488

Copy link
Copy Markdown
Author

Thanks for pointing that out — I hadn’t caught that #2576 was already covering the exploitation scenario.

That explains the overlap, so no issue with closing this one. I appreciate you flagging the existing work rather than letting a duplicate contribution move forward.

I’ll take the existing implementation in #2576 as the canonical direction and look for non-overlapping contribution opportunities in PyRIT going forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT: Add Garak exploitation scenario

2 participants