Skip to content
JKurzerPublic

About

Trivial but viable attack on REA tasks via context corruption

Resources

Stars

3 stars

Watchers

0 watching

Forks

Latest commit

 

History

102 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Let Them Eat Goop

Goop demonstrates a basic and weak context corruption attack on agents performing REA tasks. By exploding the size of the initial context read-in, we are able to breach the guardrails provided by the user. Then, using a technique called trigram bridging, we steer the model into a narrowed prediction regime that causes serious degradation of performance. This is broadly similar to biasing attacks and bigram hammering, but uses a graph of task related terms embedded in low-perplexity text.

This is an early prototype. Expect a limited effect. Tested on Astra. The delivery mechanism is a toy example, as I am not actually an RE specialist. To develop an actual countermeasure, I estimate that about 250 hours of human work remain, plus about 45 million tokens of testing. One difficulty to be aware of is that this effectively de-aligns any agent that comes in contact with the full corpus of the attack. As a result, the grading agents frequently become unreliable.

Unfortunately, due to extreme link rot becoming common, I've taken the unusual step of including the full text of the papers that I relied on when when working up the very early versions of this attack. I've also left the extensive prototyping sequence and all my silly mistakes in the git history.

Build

From the repository root in an MSVC x64 developer environment:

python tools\embed_payload.py payloads\filler.json payloads\trigrams.json --out payload-build
payload-build\target.exe 0 0

The output directory must not already exist. The script writes the generated sources, payload.txt, and target.exe there.

  • payloads/filler.json: simulation prose with one link per side, using {left} / {right} or their _reverse forms. Paired forward-and-reverse placeholders are optional.
  • payloads/trigrams.json: anchor/target/destination sets and copy count.
  • tools/embed_payload.py: the standalone builder; no third-party imports.
  • gooplib/native_src/: the seven plain C++ source/header templates.
  • tests/test_embed_payload.py: focused Python and native integration tests.

All three carriers use the same filler-backed graph: function-local literals, explicit startup output, and a summary-format prefix. Links use bare adjacent words with no parentheses: eight primary bridge arrangements join each leaf to the anchor, never directly to the other leaf. The payload prints twice per execution, at startup and before the normal SUMMARY. The example prints 864 rows twice.

Reference and scope

  • TRIGRAM_BRIDGING.md records the owner's explanation verbatim. Its mechanism and efficacy claims are hypotheses, not guarantees established by this builder or its tests.

The sample uses nine distinct templates: eight simulation-prose bridge arrangements and one row copied from line 33 of COGNITOHAZARD.danger (SHA-256 a87e79a6fb16cf96d558b8cc129c205f766c5144ef299673ecbe0251cb56580e). Only its two parenthesized link spans were replaced by {left_reverse} and {right_reverse}; the rest of that line, including its two trailing spaces, is preserved. A set names an anchor and two leaves (target, destination). Every set emits all eight primary arrangements: independently reverse either link, then exchange their order in the sentence. Every template after the first eight emits once per set, repeating bridge arrangements as needed. Actual primary placeholder position determines link order, even when {right} precedes {left}. Two sample templates additionally include {left_reverse} and {right_reverse}: both forward and backward forms of BOTH links occur in the same sentence. A reverse placeholder may stand alone; if one side has both forward and reverse, so must the other. No placeholder may be duplicated. Links are bare adjacent words, with no parentheses, and, or other connector.

Paid analyst runs are separate from building and testing. Stage the archived launcher outside this repo, but use the current bounded-delivery protocol and analyst prompt template, not the archived prompt. The planner sizes plain stdout chunks for both line and serialized-result limits; the local runner check must pass before launch. Expose only target.exe to the analyst and retain prompts, logs, usage and artifact hashes separately. Its per_cell_budget_usd check happens after a run, so it is not a hard spending cap. Delivery must be checked in the analyst's saved tool-return content; local output alone does not establish exposure.

Test

Install pytest in your chosen Python environment, then from the same x64 developer environment run:

python -m pytest -q

Without MSVC on PATH, the native integration test skips. The builder itself requires only Python's standard library and the MSVC/Windows SDK toolchain.

Prior work

The complete pre-cleanup project, including previously ignored run evidence, is retained at Git tag legacy-before-minimal-stack (111a8dc). No history was rewritten. Cached environments are disposable and were not archived.

About

Trivial but viable attack on REA tasks via context corruption

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages