Skip to content

ci: automate and harden doc-gardening (model pin + per-PR advisory) #913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Roasbeef merged 7 commits into from
Aug 7, 2026
Merged

ci: automate and harden doc-gardening (model pin + per-PR advisory) #913

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
88b2b3b
ci: pin nightly doc-gardening to opus and harden its prompt
Roasbeef Jul 10, 2026
4f1100e
ci: add per-PR doc-gardening advisory
Roasbeef Jul 10, 2026
5bb9818
ci: open the nightly failure canary with the workflow token
Roasbeef Jul 16, 2026
23bf850
ci: bump doc-gardening to opus 5
Roasbeef Aug 4, 2026
72504cf
ci: drop unused shell-spawning tools from doc-gardening allowlist
Roasbeef Aug 4, 2026
338513e
ci: harden doc-gardening per adversarial review
Roasbeef Aug 4, 2026
2e32546
ci: fix doc-gardening scope and scrub bugs found in review
Roasbeef Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 72 additions & 15 deletions .github/workflows/doc-gardening-nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@ permissions:
# only needs read access for the action's own bookkeeping.
contents: read
pull-requests: read
# The failure canary opens its tracking issue with the workflow's own
# GITHUB_TOKEN rather than BOT_GITHUB_TOKEN, so an expired or revoked
# bot token can never take the canary down with it (that exact failure
# mode went unnoticed for two days when the bot PAT hit its 90-day
# expiry: checkout died AND the notify step died with the same 401).
issues: write

jobs:
sweep:
Expand All @@ -33,7 +39,7 @@ jobs:
timeout-minutes: 60
steps:
- name: Checkout (bot token)
uses: actions/checkout@v5
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
with:
# Use BOT_GITHUB_TOKEN so subsequent `git push` operations
# authenticate as the bot. This is required for the resulting PR
Expand All @@ -46,8 +52,19 @@ jobs:
# returns empty history and turns the sweep into a false no-op.
fetch-depth: 0

- name: Neutralize git hooks
# Point core.hooksPath at an empty target so no repository hook can
# execute during this run. The model is granted native Write and
# Bash(git commit:*), so a prompt injection could otherwise drop a
# .git/hooks/pre-commit and have the sweep's own `git commit` run
# it with the bot token in env. `git config` is deliberately NOT in
# the model's allowlist, and `git -c core.hooksPath=... commit` does
# not match the Bash(git commit:*) prefix, so the model cannot undo
# this from inside the action.
run: git config core.hooksPath /dev/null

- name: Set up Go
uses: actions/setup-go@v5
uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5
with:
go-version-file: go.mod
cache: true
Expand Down Expand Up @@ -77,7 +94,7 @@ jobs:
done

- name: Run /doc-gardening sweep and open PR
uses: anthropics/claude-code-action@v1
uses: anthropics/claude-code-action@fb326f7c687fa273d0843a59bbd2ddbc9e481ca1 # v1
env:
# Belt-and-suspenders: also surface the bot token as GH_TOKEN on
# the step env so `gh` inside Claude's Bash subprocess picks it
Expand Down Expand Up @@ -115,11 +132,16 @@ jobs:
# the Actions log so we can grep what the model actually did
# on a NO-OP day. Without this, the action buffers stdout
# internally and the Actions log shows only step start/end.
# GitHub auto-masks `secrets.*` values, so BOT_GITHUB_TOKEN
# and CLAUDE_CODE_OAUTH_TOKEN remain redacted; the residual
# leak risk is content Claude reads off disk that isn't
# registered as a secret. For a docs-only sweep on a public
# repo, that risk is acceptable.
# GitHub auto-masks `secrets.*` values in the log, so
# BOT_GITHUB_TOKEN and CLAUDE_CODE_OAUTH_TOKEN stay redacted
# there. Two residual risks remain on this private repo: content
# Claude reads off disk that isn't registered as a secret, and
# the base64-encoded credential actions/checkout persists into
# .git/config (a different string than the masked secret, so a
# `cat .git/config` would print it unredacted). Both are bounded
# by keeping BOT_GITHUB_TOKEN a fine-grained PAT scoped to only
# these repos; log readers are first-party. Accepted on that
# basis.
show_full_output: "true"
prompt: |
You are running in a nightly GitHub Actions cron job. Your job
Expand Down Expand Up @@ -165,9 +187,31 @@ jobs:
makes the sweep actually look at everything that changed
since the last merged nightly (not just "new packages").

2. **Validate.** Phase 7 runs `make doc-check`. If it fails,
iterate to fix the errors — never commit broken docs. If you
cannot make it pass after reasonable effort, print
2. **Mirror and scrub, then validate.** Before running
`make doc-check`, do two things that the check itself will
NOT catch on its own:

- **Mirror every CLAUDE.md to its AGENTS.md.** For each
`CLAUDE.md` you created or edited, `cp` it to the sibling
`AGENTS.md` so the pair is byte-identical. The root
`CLAUDE.md`/`AGENTS.md` pair is the one most often missed —
mirror it too. `make doc-check` verifies byte-identity, so
a missed mirror fails the build; do the copy proactively.
- **Scrub tool/formatting artifacts.** Fan-out doc edits can
leak stray tags into `.md` files (e.g. a literal
`</content>` or `</invoke>`, or a trailing
` ``` ` fence with no opener). Run
`grep -rn --include='*.md' -e '</content>' -e '</invoke>'
-e '<parameter' .` over the changed docs (scoped to
markdown so a legitimate occurrence in Go source or a test
fixture is never treated as an artifact) and delete any
such artifact before
committing. A doc with a leaked tag is a failed sweep even
if `make doc-check` is green.

Then run `make doc-check`. If it fails, iterate to fix the
errors — never commit broken docs. If you cannot make it
pass after reasonable effort, print
`DOC-GARDENING ABORTED: make doc-check still failing` and
exit without committing, pushing, or opening a PR.

Expand All @@ -179,7 +223,11 @@ jobs:
- Create and check out a fresh branch:
`BRANCH="doc-gardening/nightly-$(date -u +%Y-%m-%d)-${RUN_NUMBER}"`
then `git checkout -b "$BRANCH"`.
- Stage everything with `git add -A`.
- Stage only markdown docs with `git add -- '*.md'`. Never
`git add -A`: this sweep is docs-only, so scoping the add
to `*.md` guarantees a stray binary or any secret-bearing
file dropped into the worktree can never be committed or
pushed.
- Write the commit message to `.git/COMMIT_MSG` using this
template (substitute today's UTC date):

Expand Down Expand Up @@ -220,6 +268,10 @@ jobs:
- Never push to `main` directly.
- Never modify files outside this repo.
- Never add reviewers or assignees to the PR.
- This sweep is docs-only. Never run `go build`, `go test`, or
anything that compiles the tree — those drop stray binaries
(`accounting`, etc.) into package dirs that then get swept
into `git add -A`. Use `go doc` for symbol lookups only.
- If any `gh` command fails, print the error and exit non-zero
so the workflow's failure handler opens an issue.
# Tool capability boundary for Claude.
Expand All @@ -243,16 +295,21 @@ jobs:
# gh operations even with the native edit tools granted.
claude_args: >-
--max-turns 200
--model claude-opus-5
--allowed-tools
"Edit,Write,MultiEdit,Bash(git status:*),Bash(git diff:*),Bash(git add:*),Bash(git commit:*),Bash(git checkout:*),Bash(git log:*),Bash(git rev-parse:*),Bash(git branch:*),Bash(git fetch:*),Bash(git push:*),Bash(make doc-check:*),Bash(go doc:*),Bash(find:*),Bash(ls:*),Bash(cp:*),Bash(cat:*),Bash(head:*),Bash(tail:*),Bash(date:*),Bash(sort:*),Bash(sed:*),Bash(awk:*),Bash(gh pr create:*),Bash(gh pr list:*),Bash(gh pr view:*),Bash(gh auth status:*)"
"Edit,Write,MultiEdit,Bash(git status:*),Bash(git diff:*),Bash(git add:*),Bash(git commit:*),Bash(git checkout:*),Bash(git log:*),Bash(git rev-parse:*),Bash(git branch:*),Bash(git fetch:*),Bash(git push:*),Bash(make doc-check:*),Bash(go doc:*),Bash(grep:*),Bash(ls:*),Bash(cp:*),Bash(cat:*),Bash(head:*),Bash(tail:*),Bash(date:*),Bash(sort:*),Bash(gh pr create:*),Bash(gh pr list:*),Bash(gh pr view:*),Bash(gh auth status:*)"
--disallowed-tools
"Bash(rm:*),Bash(rmdir:*),Bash(git push --force:*),Bash(git push -f:*),Bash(git push --delete:*),Bash(git reset --hard:*),Bash(git commit --amend:*),Bash(gh api:*),Bash(gh secret:*),Bash(gh auth login:*),Bash(gh auth logout:*),Bash(gh auth refresh:*),Bash(gh repo delete:*),Bash(gh workflow:*),Bash(gh release:*),Bash(curl:*),Bash(wget:*)"

- name: Notify on failure (create tracking issue)
if: failure()
id: notify_create
env:
GH_TOKEN: ${{ secrets.BOT_GITHUB_TOKEN }}
# Deliberately NOT the bot token: the most likely cause of a
# failed run is the bot token itself going bad, and a canary
# that authenticates with the thing it is monitoring reports
# nothing exactly when it matters most.
GH_TOKEN: ${{ github.token }}
run: |
# Create the tracking issue WITHOUT labels first so a missing or
# renamed label can never swallow the canary. The label-attach
Expand All @@ -267,7 +324,7 @@ jobs:
if: failure() && steps.notify_create.outputs.issue_url != ''
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.BOT_GITHUB_TOKEN }}
GH_TOKEN: ${{ github.token }}
ISSUE_URL: ${{ steps.notify_create.outputs.issue_url }}
run: |
# Best-effort label attach. `continue-on-error: true` plus `|| true`
Expand Down
175 changes: 175 additions & 0 deletions .github/workflows/doc-gardening-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
name: Doc Gardening (PR advisory)

# Per-PR companion to the nightly doc-gardening cron. When a PR touches Go
# package code, this runs the /doc-gardening skill scoped to ONLY the changed
# packages and posts the proposed CLAUDE.md/AGENTS.md diff back as a single
# sticky PR comment. It never pushes, never opens a PR, and never fails the
# build — the comment is the entire signal, so a stale doc is surfaced in
# lockstep with the change that caused it instead of waiting up to 24h for the
# nightly. Flipping it from advisory to a (non-required) soft-fail is a
# one-line change; see the ADVISORY toggle near the end of the prompt.

on:
pull_request:
types: [opened, synchronize, reopened]
paths:
# Only Go source changes can drift a package's docs. Docs-only PRs
# (including the nightly's own) never trigger this.
- "**/*.go"

concurrency:
# One run per PR; a new push supersedes the in-flight advisory.
group: doc-gardening-pr-${{ github.event.pull_request.number }}
cancel-in-progress: true

permissions:
contents: read
# Needed so the bot can post/update its advisory comment.
pull-requests: write

jobs:
advise:
name: Scoped doc drift advisory
runs-on: ubuntu-latest
timeout-minutes: 30
# Skip fork PRs (no access to the OAuth/bot secrets) and the nightly's
# own automation PRs (already a full sweep). Same-repo branches only.
if: >-
github.event.pull_request.head.repo.full_name == github.repository &&
!contains(github.event.pull_request.labels.*.name, 'automation')
steps:
- name: Checkout (bot token)
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
with:
token: ${{ secrets.BOT_GITHUB_TOKEN }}
# Full history + the PR head so the skill can diff base..head to
# find changed packages. A shallow clone breaks that diff.
fetch-depth: 0
ref: ${{ github.event.pull_request.head.sha }}

- name: Set up Go
uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5
with:
go-version-file: go.mod
cache: true

- name: Prime module cache
run: go mod download

- name: Run scoped /doc-gardening and post advisory
uses: anthropics/claude-code-action@fb326f7c687fa273d0843a59bbd2ddbc9e481ca1 # v1
env:
GH_TOKEN: ${{ secrets.BOT_GITHUB_TOKEN }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
PR_NUMBER: ${{ github.event.pull_request.number }}
REPO_NAME: ${{ github.repository }}
RUN_ID: ${{ github.run_id }}
SERVER_URL: ${{ github.server_url }}
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
github_token: ${{ secrets.BOT_GITHUB_TOKEN }}
show_full_output: "true"
prompt: |
You are running as a per-PR documentation advisor in GitHub
Actions. Your job is to check whether this PR's code changes have
left any per-package `CLAUDE.md` doc stale, and if so, post a
single advisory comment with the proposed fix. You do NOT commit,
push, or open a PR. The `gh` CLI is authenticated as the bot.

Do not ask for confirmation. Execute in order:

1. **Find the changed packages.** Run:

git diff --name-only "$BASE_SHA"..."$HEAD_SHA" -- '*.go' \
| xargs -r -n1 dirname | sort -u

For each directory in that list, keep only the ones that
already contain a `CLAUDE.md` (packages without one are the
nightly full-sweep's job, not this advisory's — do not create
new docs here). Call this the SCOPE set. Print it as
`SCOPE: <space-separated dirs>`. If the SCOPE set is empty,
print `DOC-PR NO-OP: no documented packages changed` and stop
without commenting.

2. **Audit each scoped package.** Read
`.claude/skills/doc-gardening/SKILL.md` and apply its
update logic (Phases 2-6) to ONLY the packages in SCOPE.
For each, diff `"$BASE_SHA"..."$HEAD_SHA"` over that package's
`.go` files and reconcile its `CLAUDE.md` against the current
code: new/removed/renamed exported types, changed CLI flags or
config, new invariants, changed dependency relationships. Edit
the `CLAUDE.md` in place. After editing each one, `cp` it to
the sibling `AGENTS.md` so the pair stays byte-identical.

3. **Self-check.** Run `make doc-check`. If it fails, fix the
errors before proposing anything. Also grep the changed docs
for leaked artifacts
(`grep -rn --include='*.md' -e '</content>' -e '</invoke>'
-e '<parameter' .` — scoped to markdown so a legitimate
occurrence in Go source or a test fixture is never treated
as an artifact to delete)
and remove any. Never propose a diff that would not pass
`make doc-check`.

4. **Compute the proposal.** Run `git diff -- '*.md'`. The
single-star pathspec is deliberate: git's `*` already spans
`/`, so a leading `**/` requires at least one slash and
would silently skip the root `CLAUDE.md`/`AGENTS.md` pair —
exactly the pair a root-level `.go` change puts in SCOPE.
If the diff is empty, print
`DOC-PR NO-OP: scoped docs already current` and stop without
commenting.

5. **Post ONE sticky advisory comment.** Write the comment body
to `.git/DOC_ADVISORY.md`. It must start with the exact hidden
marker line `<!-- doc-gardening-pr -->` so the comment is
idempotent, then contain:
- A one-sentence summary naming the packages in SCOPE.
- The proposed changes inside a single ```diff fenced block
(the output of the `git diff` above).
- A short "How to apply" note:
`git apply` the diff, or run the nightly skill locally.
- A footer line linking the run:
`${SERVER_URL}/${REPO_NAME}/actions/runs/${RUN_ID}`.

Post it, replacing any prior advisory on this PR so the thread
never accumulates duplicates:

if gh pr comment "$PR_NUMBER" --edit-last \
--body-file .git/DOC_ADVISORY.md 2>/dev/null; then
echo "DOC-PR UPDATED existing advisory"
else
gh pr comment "$PR_NUMBER" \
--body-file .git/DOC_ADVISORY.md
echo "DOC-PR POSTED new advisory"
fi

6. **Finish (ADVISORY).** Print `DOC-PR ADVISORY POSTED` and exit
0 regardless of whether drift was found. This check is
advisory: the comment is the signal, and CI does not fail on
drift. To make drift a soft failure later (a red X that is NOT
a required check), change this step to exit non-zero when a
comment was posted — but leave the branch-protection required
set untouched so it never blocks merge.

Constraints:
- Never commit, push, `git add`, open a PR, or modify main.
- Never run `git push`. Never run `go build`/`go test` (they drop
stray binaries into package dirs). Use `go doc` for lookups.
- Never create new `CLAUDE.md`/`AGENTS.md` files; only update ones
that already exist for packages in SCOPE.
- Never modify files outside this repo.
- The comment body must contain ONLY the doc diff described
above. Never read `.git/config`, environment variables, or
any credential into it. `actions/checkout` persists the bot
token into `.git/config`, and a posted comment is not
log-masked, so the comment is the one egress channel that
could carry a secret off this runner.
claude_args: >-
--max-turns 120
--model claude-opus-5
--allowed-tools
"Edit,Write,MultiEdit,Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(git rev-parse:*),Bash(make doc-check:*),Bash(go doc:*),Bash(ls:*),Bash(cp:*),Bash(cat:*),Bash(head:*),Bash(tail:*),Bash(sort:*),Bash(uniq:*),Bash(grep:*),Bash(xargs:*),Bash(dirname:*),Bash(echo:*),Bash(gh pr comment:*),Bash(gh pr view:*),Bash(gh pr diff:*)"
--disallowed-tools
"Bash(rm:*),Bash(git add:*),Bash(git commit:*),Bash(git push:*),Bash(git checkout:*),Bash(git reset:*),Bash(go build:*),Bash(go test:*),Bash(gh pr create:*),Bash(gh pr merge:*),Bash(gh api:*),Bash(gh secret:*),Bash(gh workflow:*),Bash(curl:*),Bash(wget:*)"
Loading