Conversation
|
@BiBoyang Merged in 3.38.0, with one change to the shape. I reproduced the failure on the current tree before touching anything: "polish draft.md, and also write the commit message" delivered both with no scope declaration, exactly as your evals reported. After the change the same request names the commit message as outside the skill and returns only the prose. My first instinct was that the per-skill lines duplicated table row 32 and should be dropped. Checking the install paths showed the opposite: Each skill states its own boundary rather than repeating one sentence, and think's routing moved from Gotchas into Hard Rules, since the failure you found was on the exact trigger phrase that table row already named. Thanks for the evals; the reproduction is what made this easy to accept. |
Summary
Every skill declares
Not for ...in its frontmatter, but that declaration lives at the routing layer: it decides whether the skill gets loaded, not what the model does once loaded. When a request bundles an out-of-scope item with in-scope work ("polish this, and translate it while you're at it"), the skill is loaded, the rule is in context — and the model silently takes the whole job.Verified with skill-up evals (kimi engine, with/without-skill benchmark, judge evidence from transcripts; harness + full report on the
evals/skill-up-evidencebranch of the source fork). Five of eight skills failed their boundary case; the two that held (learn, ui) are exactly the ones whose boundary rule lives in the body as a behavioral instruction.Evidence (with_skill runs, skill confirmed loaded via transcript)
fix: release lock properly..., zero scope declarationbuggy.pyand ran it — breaking its own report-only discipline/writerouting had no fallback when/writeisn't installedwrite_pdf()and generated report.pdfThe think case is the most instructive: the rule existed, was precisely matched by the prompt, and still didn't fire — a passive descriptive row at the bottom of a Gotcha table doesn't carry enough weight in a live conversation. The ui case proves the fix works: a body-level behavioral rule holds.
Changes
rules/anti-patterns.md: new row 32 "Silent scope creep on bundled asks" (nearest neighbor fix: use root source path to fix skill registration #18 covers silently dropping parts of a multi-ask, not silently accepting out-of-scope ones)write/healthHard Rules: bundled out-of-scope asks get named in one line, in-scope part still completed; health's also reiterates that a bundled ask never authorizes touching project filescheckDocument Review: add the missing fallback when/writeis not installedthink: promote the "判断一下这个报错" Gotcha row into a Hard Rule phrased as an action (route out in one line before doing anything else); the Gotcha row is removed so the rule lives in exactly one placehuntHard Rules: a bundled "while you're at it, add X" is named and deferred; user agreement to fix a surfaced bug lifts the listing restriction, it does not expand a hunt into feature workpython3 scripts/verify_skills.pypasses. No frontmatter/VERSION changes, so no regenerate needed.