Repository navigation
examples: Promptfoo red-team plugin for agent-internal privacy leakage - #32
Merged
Merged
Conversation
…cy leakage Custom file-based plugin (generator + grader) built on the AgentLeak taxonomy: 7 internal pathways, L1-L4 severity. Promptfoo's built-in privacy plugins test what a model tells the user; this one targets what agents tell each other (68.8% inter-agent leakage vs 27.2% final outputs, IEEE Access 2026, DOI 10.1109/ACCESS.2026.3704541).
nikhilred2619
pushed a commit
to nikhilred2619/agentleak
that referenced
this pull request
Nov 3, 2026
…cy leakage (yagobski#32) Custom file-based plugin (generator + grader) built on the AgentLeak taxonomy: 7 internal pathways, L1-L4 severity. Promptfoo's built-in privacy plugins test what a model tells the user; this one targets what agents tell each other (68.8% inter-agent leakage vs 27.2% final outputs, IEEE Access 2026, DOI 10.1109/ACCESS.2026.3704541).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a ready-to-use Promptfoo custom red-team plugin template under
examples/promptfoo/.Why. Promptfoo's built-in privacy plugins (
harmful:privacy,pii:*) test what a model tells the user. AgentLeak measures what agents tell each other — 68.8% leakage in inter-agent messages vs 27.2% in final outputs, with output-only audits missing 41.7% (IEEE Access, vol. 14, 2026, DOI 10.1109/ACCESS.2026.3704541).What.
agentleak-privacy-redteam-plugin.yaml— a file-based custom plugin (generator + grader) following the paper's taxonomy (7 internal pathways) and L1–L4 severity scale. Users drop it into theirpromptfooconfig.yamltoday:Companion proposal to the Promptfoo maintainers: promptfoo/promptfoo#11946