feat(ci): document changes as they merge instead of per release - #7181
Open
Haroenv wants to merge 1 commit into
Open
feat(ci): document changes as they merge instead of per release#7181Haroenv wants to merge 1 commit into
Haroenv wants to merge 1 commit into
Conversation
Up to standards ✅🟢 Issues
|
More templates
algoliasearch-helper
instantsearch-ui-components
instantsearch.css
instantsearch.js
react-instantsearch
react-instantsearch-core
react-instantsearch-nextjs
react-instantsearch-router-nextjs
vue-instantsearch
commit: |
|
Size Change: 0 B Total Size: 1.23 MB ℹ️ View Unchanged
|
Haroenv
force-pushed
the
feat/docs-automation-per-change
branch
4 times, most recently
from
August 19, 2026 14:28
d2c6529 to
61d0287
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Moves documentation automation from release-sized runs to incremental staging, followed by a release sweep and draft PR.
Changes:
- Adds per-change documentation generation on a shared staging branch.
- Introduces a reusable agent action with verification, continuation, and telemetry.
- Adds staging, guide-impact, and commit-management scripts.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
CLAUDE.md |
Documents cross-repository agent conventions. |
.github/workflows/docs-feature.yml |
Adds per-change documentation workflow. |
.github/workflows/docs-automation.yml |
Refactors release documentation sweep and PR creation. |
.github/scripts/docs-staging-commit.sh |
Commits agent output to staging. |
.github/scripts/docs-staging-checkout.sh |
Manages staging branch lifecycle. |
.github/scripts/docs-guide-impact.sh |
Ranks potentially affected documentation pages. |
.github/prompts/docs-release.md |
Defines release-sweep instructions. |
.github/prompts/docs-feature.md |
Defines per-change documentation instructions. |
.github/prompts/docs-automation.md |
Removes the previous monolithic prompt. |
.github/actions/docs-agent/prompts/fixup.md |
Adds verification fix-up instructions. |
.github/actions/docs-agent/prompts/continue.md |
Adds continuation instructions. |
.github/actions/docs-agent/action.yml |
Implements agent execution, checks, and telemetry. |
Suppressed comments (1)
.github/workflows/docs-automation.yml:222
- This direct PR-title fallback is also 74 characters, exceeding the docs repository's 70-character title limit. Shorten it just like the commit fallback.
TITLE="fix(ui-libraries): update the InstantSearch documentation for this release"
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Haroenv
force-pushed
the
feat/docs-automation-per-change
branch
from
August 19, 2026 15:35
c10d6d8 to
e8c2a45
Compare
Haroenv
marked this pull request as ready for review
August 19, 2026 15:47
Haroenv
requested review from
a team,
FabienMotte and
shaejaz
and removed request for
a team
August 19, 2026 15:47
Haroenv
force-pushed
the
feat/docs-automation-per-change
branch
from
August 19, 2026 15:53
e8c2a45 to
d752d37
Compare
shaejaz
reviewed
Aug 19, 2026
shaejaz
left a comment
shaejaz
left a comment
Contributor
There was a problem hiding this comment.
I might have missed this if it's already mentioned, but maybe once a release is done for the PR that had been merged, it can leave a comment on the docs PR that it should no longer be blocked.
Might be a bit complicated to do, but would be helpful in keeping track.
The docs automation ran once per release, in a single 50-turn Claude call. It hit the turn limit regularly (run 32247853114: `error_max_turns` at turn 51, 5m39s of a 15-minute timeout, $3.24) and the downstream steps were gated on that step succeeding, so every completed edit was discarded. Split the work by change instead of by release. `docs-feature.yml` runs on each merged `feat`/`fix` while its diff is small and unambiguous, and accumulates onto a long-lived staging branch in docs-new. `docs-automation.yml` now sweeps that branch for gaps at release time and opens one draft pull request. Both serialise on one concurrency group. Turn pressure comes from batching a week of releases into one prompt; this removes the batching. docs-new is shared across Algolia products, so the staging branch is namespaced - `instantsearch/staging`, not something generic like `docs/next` that another team's automation would collide with. Both workflows read it from a single `DOCS_BRANCH_NAME` env value. Both use a new `docs-agent` composite action, which: - Runs Claude with docs-new as the working directory, so that repository's `.claude/CLAUDE.md` -> `AGENTS.md` -> `style-guide.md` chain and its skills load. They never did before: Claude ran from the workspace root, and a CLAUDE.md in a subdirectory is not picked up. Our prompts now only carry what is InstantSearch-specific. - Verifies the result against docs-new's own checks - Vale on changed files, JSX tag balance, new pages registered in `docs.json`, links - then runs one fix-up pass with the failures and re-verifies. `AGENTS.md` calls Vale and link checks mandatory before opening a pull request; neither ran, because the old `--allowedTools` patterns used `Bash(git diff *)` rather than the `Bash(git diff:*)` prefix form and matched nothing. Three of four Bash calls in the last run were denied. - Removes DOCS_REPO_PAT from `docs-new/.git/config` for the duration of the run and restores it after. The clone URL embeds the token, the agent can read it, and anything it writes gets committed and pushed - and its prompt now contains contributor-authored pull request titles from a public repository. - Continues across passes rather than failing: an exhausted turn budget becomes a checkpoint in `PROGRESS.md`, and a partial result still ships as a draft pull request marked incomplete. - Reports turns, cost, duration and denied tool calls per pass to the step summary, so the budget can be tuned from data. Guides get the same treatment as reference pages. A new option means a new row in a reference table, which is mechanical; the failure mode that costs readers is a guide that walks through a flow the change just altered and still describes the old one - plausible, confident, and invisible because nothing is missing. `docs-guide-impact.sh` derives search terms from the changed source paths, greps the docs tree, and appends a ranked list of the pages that mention them, so "check the guides" becomes a bounded checklist. Terms that match more than 40 pages are dropped as too generic to identify anything - without that filter, `open` and `display` matched a third of the repository. Both prompts now treat guide staleness as a first-class obligation, with a deliberately high bar for creating a guide and a ban on restructuring existing ones unattended; the release sweep additionally does the cycle-level check no per-change run can, since only it sees that three separate changes have collectively made a walkthrough wrong. While mapping that surface: the flavor convention in the prompt was wrong, inherited from the original. There are no `*.js.mdx` files in docs-new - the flavor is the file name, `doc/api-reference/widgets/chat/js.mdx`, and navigation entries are the path without the extension. Both prompts now describe the real layout and tell the agent to leave other platforms' pages alone. Rebased onto the App-token change (#7182) and adopted it throughout. Both workflows mint a scoped installation token with actions/create-github-app-token, check docs-new out with actions/checkout, and author commits as the App's bot user; no DOCS_REPO_PAT remains. The scripts take a DOCS_TOKEN instead, and since actions/checkout persists its credential as an http.<url>.extraheader in .git/config - which the agent can read - the prepare script strips that after the branch lifecycle work rather than only rewriting the remote URL. Installation tokens last an hour, so both job timeouts stay comfortably inside that. docs-staging-checkout.sh is now docs-staging-prepare.sh, since it no longer clones anything. A documentation pull request can sit in review across several releases while the staging branch keeps collecting commits, so it can come to document an API that is not published yet - merging then ships documentation ahead of the code, and nothing tracked it. Each staging commit now carries `Source-PR` and `Source-Commit` trailers, and at release time `docs-release-status.sh` splits the branch into what is published and what is not, then maintains one self-updating comment on the documentation pull request saying whether it is still blocked. It also carries a queryable `unreleased` label, and promotes the pull request out of draft once everything on it is released, the sweep finished and the checks are clean - promote only, never demote, since a human may have taken it out of draft deliberately. Review findings, all of which were real: - The push token was still reachable. Replacing the remote URL hid it from .git/config but the agent step's own environment still carried it, and that is readable from /proc/self/environ. The token now never enters that step: the checkout script leaves the remote tokenless and the commit script supplies it for the push alone, restoring the tokenless URL on a trap so the failure path is covered too. - Squash-merge detection was broken by construction. docs-new squash-merges, which rewrites the commit, so the staging tip is never an ancestor of the default branch afterwards and the ancestor test concluded "still open" forever, dragging merged content into every later cycle. It now asks GitHub whether the branch's head was merged, rebuilds from the default branch, and replays anything pushed after the merge. The local test only passed before because it simulated a fast-forward. - The path boundary only warned, then `git add -A` committed the violation anyway. It now refuses to push, and stages the content paths explicitly. - A zero exit code was treated as completion. PROGRESS.md's marker is the contract; a model that stops early exits zero too. A clean exit without the marker now ends the loop without claiming the work is done. - A failed `generate:flavors` left the run reporting clean checks even though the navigation could be stale. It is a check failure now. - Verification skipped everything already committed: it ran only when the working tree was dirty, and used revision-less diffs. The release sweep could therefore open a pull request over content nothing had checked. It now covers `origin/<default>...HEAD` as well as the working tree. - The fallback title was 74 characters, breaking the 70-character rule it exists to enforce. Shortened, and the fallback is now length-checked too. Commit and pull request titles now follow docs-new's convention - `feat`/`fix`/`chore`, sentence case, no trailing period, under 70 characters. `AGENTS.md` bans `docs:` there, and pull requests squash-merge, so the old hardcoded `docs: ${TITLE}` landed a malformed commit on their default branch every time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Haroenv
force-pushed
the
feat/docs-automation-per-change
branch
from
August 19, 2026 20:02
d752d37 to
b422232
Compare
Contributor
Author
|
Good catch, and it was a real hole rather than something already covered. The docs PR only opens at release time, so at that instant everything on it is published. But the staging branch keeps accumulating while the PR sits in review, so by the time someone looks at it, it can have grown commits documenting features from after that release — and merging then ships docs for an API nobody can install. Nothing tracked that. Implemented in b422232:
Not as complicated as feared — the trailers made it mostly bookkeeping. Tested against a fixture branch covering blocked, unblocked, unblocked-but-checks-failing, already-ready, and the sweep's own untagged commit. — via Claude |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The docs automation ran once per release, as a single 50-turn Claude call. It hit the turn limit regularly and threw away the work when it did — run 32247853114 ended with:
Turns were the binding constraint, not time or money — it used 37% of its timeout budget. But the more expensive problem was that the step exited 1, so Create branch and Create PR were skipped: seven correct edits to
docs-new/doc/api-reference/widgets/chat/react.mdx, pushed nowhere.Looking at the target repository turned up more.
algolia/docs-newshipsAGENTS.md,.claude/CLAUDE.md,.claude/skills/update-doc/,style-guide.mdand.vale.ini— a full authoring contract that this workflow never loaded, because Claude ran from the workspace root and aCLAUDE.mdin a subdirectory isn't picked up.What changed
Split the work by change instead of by release.
docs-feature.ymlruns on each mergedfeat/fixwhile its diff is small and unambiguous, and accumulates onto a long-lived staging branch in docs-new.docs-automation.ymlnow sweeps that branch for gaps at release time and opens one draft PR. Both serialise on a single concurrency group.docs-new is shared across Algolia products, so the branch is namespaced —
instantsearch/staging, not something generic likedocs/nextthat another team's automation would collide with. Both workflows read it from oneDOCS_BRANCH_NAMEenv value, so the docs team can veto the name with a one-line change. Turn pressure came from batching a week of releases into one prompt — this removes the batching. It also matches docs-new's ownupdate-docskill, which is per-PR, and their "Cover one concern per PR" rule, which a release batch structurally can't satisfy.Both workflows use a new
docs-agentcomposite action:--add-dirfor the InstantSearch source), so their.claude/CLAUDE.md → @../AGENTS.md → @../style-guide.mdchain and skills load. Our prompts shrank to what's genuinely InstantSearch-specific — the cross-flavor changelog trap, flavor mapping, checkpointing.--minAlertLevel=warning, JSX tag balance, new pages checked againstdocs.json/config/*.json, links filtered to touched files. Failures feed one fix-up pass, then re-verify.generate:flavorsruns after.Bash(git diff *)— space-star, not theBash(git diff:*)prefix form — so they matched nothing, includingcheck:links, which has therefore probably never run. Three of four Bash calls in the last run were denied outright.PROGRESS.md. A partial result still ships as a draft PR marked incomplete, with instructions to finish it locally.Guides, not just reference
Reference pages are the easy half: a new option means a new row in a known table. The half that rots is a guide that walks through a flow the change just altered and still describes the old one — plausible, confident, and invisible because nothing is missing. The InstantSearch guide surface is 52 topics, including a whole
going-further/chat-customization/tree that the current weekly chat work lands squarely on.docs-guide-impact.shderives search terms from the changed source paths, greps the docs tree, and appends a ranked list of pages that mention them. Terms matching more than 40 pages are dropped as too generic to identify anything — without that filteropenanddisplaymatched 690 of 1958 pages; with it, the real release diff yields 12:Both prompts now treat guide staleness as a first-class obligation, with a deliberately high bar for creating a guide (a new widget, or a capability with nowhere to live) and a ban on restructuring existing guides unattended — those get proposed under a
Flagged for humansheading instead. The release sweep additionally does the cycle-level check no per-change run can: only it can see that three separate changes have collectively made a walkthrough wrong.Also found while mapping that surface: the flavor convention in the prompt was wrong, inherited from the original. There are no
*.js.mdxfiles in docs-new — the flavor is the file name (doc/api-reference/widgets/chat/js.mdx), and navigation entries are the path without the extension (/doc/api-reference/widgets/chat/js). Both prompts now describe the real layout and leave other platforms' pages (android.mdx,flutter.mdx, …) alone.Security. The clone URL embeds
DOCS_REPO_PATindocs-new/.git/config; the agent can read it, and anything it writes gets committed and pushed. Its prompt now also contains contributor-authored PR titles from a public repo. The action strips the token from the remote for the duration of the run and restores it via anEXITtrap.Commit convention.
AGENTS.mdsays "This repo is docs-only, sodocs:carries no information. Never use it" — and PRs squash-merge there, so the old hardcodeddocs: ${TITLE}landed a malformed commit on their default branch every time. Titles are now validated against their rule (feat/fix/chore, optional lowercase scope, sentence case, no trailing period, ≤70 chars) with a fallback when the model's title doesn't conform.Rebased onto the App-token change (#7182)
That landed on the same file this PR rewrites, so it's adopted rather than merged around. Both workflows now mint a scoped installation token with
actions/create-github-app-token, check docs-new out withactions/checkout, and author commits as the App's bot user — noDOCS_REPO_PATanywhere.It interacts with the credential-hiding fix below.
actions/checkoutpersists its credential as anhttp.<url>.extraheaderin.git/config, which the agent can read, so stripping the remote URL is no longer sufficient — the prepare script unsets that header after the branch lifecycle work, and the commit script restores credentials for the push alone. Installation tokens last an hour and both job timeouts stay inside that.docs-staging-checkout.sh→docs-staging-prepare.sh, since it no longer clones anything.Release blocking (@shaejaz's point)
A docs PR only opens at release time, so at that instant everything on it is published. But the staging branch keeps accumulating while the PR waits for review, so it can grow commits documenting features from after that release — and merging then ships docs for an API nobody can install. Nothing tracked that.
Each staging commit now carries
Source-PR/Source-Committrailers. At release time the workflow extracts the PR numbers the release publishes from the changelogs, splits the branch into published and not, and maintains one self-updating comment on the docs PR (found by an HTML marker, so it's edited rather than re-posted every release):Once the next release publishes the rest, the same comment flips to
✅ No longer blocked. There's also anunreleasedlabel so the state is filterable rather than only prose, and the PR is promoted out of draft once everything on it is released and the sweep finished and checks pass — matching docs-new's rule about draft-until-checks-green. Promote only, never demote: a human taking it out of draft deliberately shouldn't be undone.Review findings
All seven were real; two were things my own tests should have caught.
.git/config, but the agent step's own environment still carried it — readable from/proc/self/environ.trap, so the failure path is covered too.git add -Acommitted the violation anyway.PROGRESS.md's marker is the contract; a model that stops early exits zero too.generate:flavorsleft the run reporting clean checks with possibly-stale navigation.origin/<default>...HEADas well as the working tree.On the squash-merge one: my lifecycle test passed only because I simulated a fast-forward merge rather than the squash the repo actually uses. Reproducing it with
git commit-treeshows the old check plainly returning "not merged".Testing
No CI covers these paths, so everything was exercised locally:
claude/npm/npxagainst a throwaway docs repo. Injected an unbalanced<Steps>and an unregistered new page; both checks fired, the fix-up pass ran, re-verification came back clean, and the PAT was confirmed absent during the run and restored after.docs:prefix, Title Case, trailing period, and a 71-character title all fall back; the conventional one passes through.chat-customizationguides ranked at the top; empty input exits cleanly.PATCHed not re-posted, label removed, promoted), unblocked-but-checks-failing (no promotion), already-ready (no redundant call), and the sweep's own commit with no provenance trailer (counted as released). Changelog PR-number extraction checked against the realinstantsearch.jschangelog — 7 pull requests, correctly excluding thecloses [#6880]issue ref.extraheadercredential, a stubbedghfor the default branch / bot user / merged-PR lookups. Confirms the credential count goes1 → 0across prepare, the commit is authoredfx-bot[bot] <12345+fx-bot[bot]@users.noreply.github.com>, nothing is left behind after the push, and squash detection still fires.incomplete;generate:flavorsfailure reportingissues; the boundary check refusing to push a file outsidedoc/; commit-and-push against a throwaway remote confirming the remote is left tokenless afterwards.git add -A -- <paths>aborts entirely if any listed path is absent from the repo, so nothing was staged. It stages per path now.bash -n+shellcheck -S warningon everyrun:block and both scripts;js-yamlparse on all three YAML files.Two bugs found and fixed while testing:
git diffcan't see untracked files, so new pages escaped both the tag and navigation checks — the navigation check only ever fires for new pages, so it was dead code (fixed withgit add -A -N); andgit … | headunderpipefailSIGPIPEs into a failed step.Rollout notes
git merge-base --is-ancestor). All three paths tested functionally against a throwaway bare remote.CLAUDE.mdgains one line, inConventions: a cross-repo agent workflow must run with cwd inside the target repo, and--allowedToolsneeds theBash(cmd:*)prefix form. Those two mistakes account for four of the six defects here.workflow_dispatchondocs-automation.ymlwithcreate_pr: falsebefore letting the push trigger run.🤖 Generated with Claude Code