Storybook: Add regression testing for documentation loss/gain

[aduth]

What problem does this address?

Recent upgrades of Storybook have been particularly painful due to the changes in documentation extraction resulting in regressions where component descriptions or props are lost from Storybook.

Examples:

• The Storybook v10 upgrade in Update Storybook to v10 with Vite builder #74396 had several regressions, fixed in components: Add displayName to the anonymous components #74716 (missing component names in code samples) and Storybook: Fix missing props from component stories #74807 (missing props)
• The Storybook v10.4 upgrade in Storybook: Upgrade Storybook to 10.4 #77382 had several self-contained regressions that were fixed before merge, including missing component descriptions, unexpected props inclusions of @ignore'd props, and incorrect component names in code samples

Since the entire point of the Storybook site is to document components, their descriptions, available options, and code snippets, these kinds of regressions are especially painful and prevent someone from understanding how to use available components.

What is your proposed solution?

Based on previous failure scenarios, ideally we'd have some sort of mechanism to detect when props, descriptions, and code samples change as the result of a pull request.

Since many of the issues we've discovered have involved specific workarounds (JSDoc placement, displayName usage, etc.), regressions can happen in any pull request, not just those which update Storybook. For example, a pull request which relocates a component's JSDoc description or wraps/unwraps forwardRef may inadvertently break documentation on Storybook, which this regression test should detect.

Open questions:

• Should this apply to manifest documentation, live Storybook site, or both? These currently use different documentation extraction tools (longer-term they will use the same)
  • If testing the live site, how does that work? Is it feasible to run headless browser testing against hundreds of Storybook pages, potentially twice if we compare it against the "live" copy directly?
• How is this surfaced? Test failure? GitHub comment?
• What kind of sign-off is required? Is a GitHub comment enough to show awareness? Something more like the CHANGELOG optional build failures?

More context in #77382 (comment) :

We have some coverage already with the design system validate-manifest.mjs script run in CI to ensure that there's at least some components with props. But all that's doing is ensuring that it's not completely broken. With the issues we've seen, I think we can get more granular with this to see when descriptions/props/components are added/lost.

We'll also want to be careful to avoid "props gained" as an objectively good measure, since what happened here was that we accidentally "gained" props which were intended to be hidden. Ideally we'd have some sort of mechanism where we highlight specific differences in a way that someone can sign off on the changes as expected.

The tricky thing is that we currently have two different docgen tools for Storybook. All signs are that Storybook is investing in this new "meta" extractor as the default for everything, but currently it's only available for the manifests. So we can test against the manifests (which is easy to do), but that doesn't provide us much guarantees about the user-facing Storybook site. And testing the user-facing Storybook site is difficult, as even with a headless browser test environment to do so, there's hundreds of pages of documentation. Given the long-term direction, it might be simplest to just test the manifest file and trust that it'll be future-proof to be representative of what a user would see in the browser, although that leaves a gap until that meta extractor is actually used for the live site.

[aduth]

Another option that came up in exploring this with my AI agent: An artifact committed to source control, similar to our documentation generated from npm run docs:build that is forced to be kept in sync through "Check local changes" step of static CI checks. This could be the entire manifest, but that's quite large. It could also be a subset of the manifest that we care about (descriptions, props, code snippets). If we did that, it might also raise a question if we should publish / use that for the MCP ourselves. At that point it's basically our own manifest.

The main advantage here over an optional failing build is that it gives someone an option to get the build green by intentionally reconciling the difference, and an easy thing to compare against (rather than sifting through git history). The downside is that it's another artifact to maintain in source control.

[aduth]

Where I'm currently leaning with this is something of a middle-ground solution:

• Committing an artifact that has a list of components and their list of props (reasonably sized, <1000 lines, possibly YAML-formatted for easy diff)
  • Follows the idea in Storybook: Add regression testing for documentation loss/gain #79415 (comment) as far as forcing this to be kept in sync similar to what we do with docs
• Assume that every component in the manifest has a description, and fail otherwise. Don't include description in the committed artifact.
  • Depends on Components: Add missing descriptions for design system components #79460
• Similarly assume that every prop has a description. There's a lot of existing gaps here. We'll need to maintain an allowlist of exceptions. But it gives us something to work toward.
• Remove / update validate-manifest.mjs script, as the updated approach makes this existing validation redundant

The only thing missing from this is code snippet regression testing. Those can be quite noisy for including in a committed artifact file. One thought is committing just a hash of the snippet contents. I'm inclined to think it's okay to leave out for an initial implementation though.