Skip to content

fix: fence off bundled out-of-scope requests instead of silently accepting them - #87

Closed
BiBoyang wants to merge 2 commits into
tw93:mainfrom
BiBoyang:fix/scope-fence
Closed

BiBoyang wants to merge 2 commits into
tw93:mainfrom
BiBoyang:fix/scope-fence

Conversation

@BiBoyang

Copy link
Copy Markdown
Contributor

Summary

Every skill declares Not for ... in its frontmatter, but that declaration lives at the routing layer: it decides whether the skill gets loaded, not what the model does once loaded. When a request bundles an out-of-scope item with in-scope work ("polish this, and translate it while you're at it"), the skill is loaded, the rule is in context — and the model silently takes the whole job.

Verified with skill-up evals (kimi engine, with/without-skill benchmark, judge evidence from transcripts; harness + full report on the evals/skill-up-evidence branch of the source fork). Five of eight skills failed their boundary case; the two that held (learn, ui) are exactly the ones whose boundary rule lives in the body as a behavioral instruction.

Evidence (with_skill runs, skill confirmed loaded via transcript)

Skill Bundled request Observed behavior
write polish + "also write the commit message" Delivered fix: release lock properly..., zero scope declaration
health config audit + "see why buggy.py fails" Edited buggy.py and ran it — breaking its own report-only discipline
check code review + "polish the release announcement" Produced two polished drafts; the body's /write routing had no fallback when /write isn't installed
hunt diagnose + "add PDF export while you're at it" Implemented write_pdf() and generated report.pdf
think "判断一下这个报错" — the exact trigger phrase in its own Gotcha row Went straight into debugging analysis; the row never fired
ui (control) landing page + "also typeset the print doc" Held: declined the print layout and handed off to Kami — because the rule is in the body

The think case is the most instructive: the rule existed, was precisely matched by the prompt, and still didn't fire — a passive descriptive row at the bottom of a Gotcha table doesn't carry enough weight in a live conversation. The ui case proves the fix works: a body-level behavioral rule holds.

Changes

  • rules/anti-patterns.md: new row 32 "Silent scope creep on bundled asks" (nearest neighbor fix: use root source path to fix skill registration #18 covers silently dropping parts of a multi-ask, not silently accepting out-of-scope ones)
  • write / health Hard Rules: bundled out-of-scope asks get named in one line, in-scope part still completed; health's also reiterates that a bundled ask never authorizes touching project files
  • check Document Review: add the missing fallback when /write is not installed
  • think: promote the "判断一下这个报错" Gotcha row into a Hard Rule phrased as an action (route out in one line before doing anything else); the Gotcha row is removed so the rule lives in exactly one place
  • hunt Hard Rules: a bundled "while you're at it, add X" is named and deferred; user agreement to fix a surfaced bug lifts the listing restriction, it does not expand a hunt into feature work

python3 scripts/verify_skills.py passes. No frontmatter/VERSION changes, so no regenerate needed.

@tw93 tw93 closed this in 76db83e Sep 19, 2026
@tw93

tw93 commented Sep 19, 2026

Copy link
Copy Markdown
Owner

@BiBoyang Merged in 3.38.0, with one change to the shape.

I reproduced the failure on the current tree before touching anything: "polish draft.md, and also write the commit message" delivered both with no scope declaration, exactly as your evals reported. After the change the same request names the commit message as outside the skill and returns only the prose.

My first instinct was that the per-skill lines duplicated table row 32 and should be dropped. Checking the install paths showed the opposite: npx skills add and the per-skill plugin entries install the skill directory alone, with no rules/, so for those users row 32 never loads. Your per-skill lines are the part that reaches everyone. AGENTS.md called that file always-on, which is not true, and it now says so.

Each skill states its own boundary rather than repeating one sentence, and think's routing moved from Gotchas into Hard Rules, since the failure you found was on the exact trigger phrase that table row already named. Thanks for the evals; the reproduction is what made this easy to accept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants