fix(web): consolidate vinext.dev's claim to its own name in search - #3062
Draft
MattieTK wants to merge 9 commits into
Draft
fix(web): consolidate vinext.dev's claim to its own name in search#3062MattieTK wants to merge 9 commits into
MattieTK wants to merge 9 commits into
Conversation
The eight example apps deployed to *.vinext.workers.dev all returned 200 with no robots directive, and four carried the project name in <title>. "vinext Benchmarks" at benchmarks.vinext.workers.dev currently outranks vinext.dev/benchmarks for its own branded query, so these deploys are competing with the canonical site for the brand term rather than supporting it. App Router examples get `noindex, nofollow` via the route they already use for head content. The two Pages Router examples have no _document, so they get a static robots.txt instead of changing the HTML those examples exist to demonstrate. The benchmarks example is also retitled off the exact term it was competing on. Claude-Session: https://claude.ai/code/session_01WbjcMmMpTWTZCVw26Nuq78
addPreviewRobotsHeader only matched the `-vinext-web.vinext.workers.dev` preview suffix, which left the production workers.dev origin indexable. Public documents there already 308 to vinext.dev, but the deliberate /api/ carve-out does not, so those JSON endpoints stayed crawlable on a host that competes with the canonical domain for the brand term. Widen the rule to any *.vinext.workers.dev host and rename the helper to say what it now does. vinext.dev itself is untouched. Claude-Session: https://claude.ai/code/session_01WbjcMmMpTWTZCVw26Nuq78
vinext.dev is outranked for its own name by community mirrors that ship richer entity markup than the official site does. Three gaps stood out. The JSON-LD lived on the home page only, with no @id, no sameAs and no SoftwareApplication, so nothing tied the site to the repo, the announcement, or the npm package. Move it to a shared graph rendered from the root layout, stably @id'd, with sameAs listing the profiles only the official site can legitimately claim, and breadcrumbs on subpages. There was no og:image anywhere, and the layout hardcoded twitter:card=summary, which suppresses the large-image upgrade. Add a 1200x630 card and switch to summary_large_image. Neither competing site has one, so this is an outright advantage rather than catch-up. Titles were hand-suffixed per page, so a new page silently inherited the home page title. Use title.template instead. Also put the brand token in the h1s, which none of them carried, and link the home page stats to the pages that substantiate them -- the body previously had no internal links at all, and dofollowed two competing workers.dev hosts. Claude-Session: https://claude.ai/code/session_01WbjcMmMpTWTZCVw26Nuq78
The project's real documentation is a 56KB README that exists only on github.com. That vacuum is what the third-party docs mirrors fill, and it leaves vinext.dev with three indexable URLs and 798 words to defend its own name with. Serving the README on the canonical domain adds ~6500 extractable words to the site. Rendered at build time inside a Vite plugin rather than at request time: the README only changes between deploys, and a runtime pipeline would cost roughly 120KB of gzipped Worker bundle plus shiki grammar compilation on every cold render. Nothing here ships to the Worker. The markdown renderer is hand-rolled because unified/remark/rehype are not declared dependencies of apps/web and adding them would need an unfrozen lockfile. It covers exactly the constructs the README uses and its output matches a reference remark pipeline element for element: 15 h2, 21 h3, 6 tables, 6 <details>, 41 code blocks, 43 heading ids. If the dependencies are added later, swapping the plugin internals is a contained change. Note that compiling the README as MDX instead is not equivalent -- it silently drops all six <details> blocks unless rehype-raw is added back, and registering @mdx-js/rollup here would disable vinext's own MDX auto-injection app-wide. No rel=canonical to github.com: that would explicitly concede the ranking. The intended end state is the inverse of today, with the README shrinking to an overview that links here for detail. Also adds the nav entry that makes the page reachable, and states in the footer that vinext.dev is the official site -- the mirror competing for the name explicitly disclaims being it, so saying so plainly is the cheapest disambiguation signal available. Claude-Session: https://claude.ai/code/session_01WbjcMmMpTWTZCVw26Nuq78
Without this route vinext falls back to a built-in 404 document, which emits a second <title> and brands every missing URL as the home page. `robots` has to be restated here rather than inherited: metadata keys are replaced rather than merged down the tree, so omitting it makes the page pick up the root layout's `index, follow` and contradict the `noindex` vinext already emits for 404s. Claude-Session: https://claude.ai/code/session_01WbjcMmMpTWTZCVw26Nuq78
The sitemap was a hardcoded three-URL literal with no lastModified, despite /compatibility and /benchmarks changing daily. Read the actual timestamps from D1 so a crawl scheduled off this file tracks the data rather than the release cadence, and add /readme. changeFrequency and priority are dropped: Google ignores both. The DB client is imported lazily and by relative specifier so the module stays unit-testable outside the Worker -- it imports cloudflare:workers, which only resolves inside the Cloudflare environment. Off-Worker the import rejects and every route falls back to the build timestamp, which is also what happens if D1 is unavailable in production. A stale lastmod is much cheaper than a sitemap that 500s. Claude-Session: https://claude.ai/code/session_01WbjcMmMpTWTZCVw26Nuq78
/compatibility shipped 513KB of HTML for 375 words of extractable text. All 799 test-file names existed only inside <svg><title> tooltips, which are accessibility labels rather than indexable page text, and the "View table" dialog that renders the same data is a Base UI portal, so it is absent from SSR HTML entirely and unmounts when closed. That data is the site's one unfakeable advantage over the mirrors competing for the name -- first-party results a scraped copy cannot invent -- and none of it was readable. Render it as a plain <table> grouped by feature, inside a native <details> so the markup is in the DOM regardless of open state. Native elements rather than the Kumo Table: that component is client-marked, so rendering ~800 rows through it would mint thousands of client references in the flight payload for no interactivity. The label vocabulary moves to a plain module first. It previously lived in a "use client" file, where importing it into the server component would turn each helper into a client-reference proxy that throws when called. Claude-Session: https://claude.ai/code/session_01WbjcMmMpTWTZCVw26Nuq78
Follows the preceding commits: the sitemap is now async and carries lastmod, benchmark titles are templated rather than hand-suffixed, and the noindex host rule covers the production workers.dev origin as well as preview aliases. Adds assertions that lastmod is present and valid on every entry, and that changeFrequency and priority stay absent so nobody reintroduces them -- Google ignores both. Claude-Session: https://claude.ai/code/session_01WbjcMmMpTWTZCVw26Nuq78
commit: |
Contributor
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
Performance benchmarksCompared 0 improved · 0 regressed · 6 within ±1.5%
View detailed results and traces 🟢 improvement · 🔴 regression · ⚫ change below 1.5% · paired base/head |
…ring sanitization' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vinext.dev is outranked for the query "vinext" by sites the project does not control. This addresses the on-site half of that.
What is actually losing the query
Two things, neither of them meta tags. The metadata from #2988 and #2989 is largely fine.
Nothing authoritative links to vinext.dev.
blog.cloudflare.com/vinext/— the highest-authority page about the project — mentions the domain zero times across 536KB, linksgithub.com/cloudflare/vinexteight times, and sends readers tobenchmarks.vinext.workers.devrather thanvinext.dev/benchmarks. GitHub renders the repo's website link asrel="nofollow", so no equity flows from there either. That part is an editorial fix on properties we own, not a code change — see "Not in this PR" below.There is nothing on the site to rank. The real documentation is a 56KB README that exists only on github.com. Three indexable URLs and 798 words were defending the name. Meanwhile the site's one unfakeable asset — 799 first-party Next.js deploy-suite results — was invisible:
/compatibilityserved 513KB of HTML for 375 words of extractable text, because every test-file name lived inside an<svg><title>tooltip.It is also worth knowing that the competition is not only
vinext.io. An auto-generated Mintlify mirror of this repo's docs runs on two hosts (cloudflare-vinext.mintlify.appandmintlify.wiki), publishes 96 self-canonical URLs titled "vinext - vinext", links back to nothing, and currently outranks both vinext.dev and vinext.io. If that project is connected to our GitHub org, one cross-canonical fixes it; if not, it is a scraped mirror using the project name.What is in here
Eight commits, reviewable in order:
fix(examples)*.vinext.workers.devdemos were indexable, four brand-titled.benchmarks.vinext.workers.dev("vinext Benchmarks") outranks our own/benchmarksfor its own query.fix(web)addPreviewRobotsHeaderonly matched preview aliases, leaving the production workers.dev origin's/api/carve-out crawlable.feat(web)metadata@id'd JSON-LD graph withsameAs, rendered site-wide, plus breadcrumbs.og:imageandsummary_large_image(neither competing site has one).title.template. Brand token in all three h1s. Internal links from the home page body, which previously had none.feat(web)/readmefeat(web)404<title>tags and branded every missing URL as the home page.feat(web)sitemaplastmodfrom D1.changefreq/prioritydropped — Google ignores both.feat(web)compatibility<table>instead of SVG tooltips.test(web)Two decisions worth a second opinion
/readmeparses its own markdown.unified/remark/rehypeare in the lockfile via fumadocs inexamples/, but none resolve fromapps/webunder pnpm's strict isolation, and declaring them needs an unfrozen install. So the page ships a ~300-line renderer covering exactly the constructs the README uses, run at build time in a Vite plugin — zero parser bytes reach the Worker. Its output matches a reference remark/rehype pipeline element for element: 15 h2, 21 h3, 6 tables, 6<details>, 41 code blocks, 43 heading ids. If someone adds the dependencies later, swapping the plugin internals is contained.Worth recording because it is the obvious alternative: compiling the README as MDX silently drops all six
<details>blocks — the entire "other platforms" deployment section — unlessrehype-rawis added back, and registering@mdx-js/rolluphere would disable vinext's own MDX auto-injection app-wide./readmeduplicates github.com, and github.com will probably win. Norel=canonicalto GitHub is set, deliberately — that concedes the ranking outright. This only pays off properly if the README later shrinks to an overview that links here for detail. Happy to drop this commit if the team would rather do it that way round from the start.Verification
Checked against a real
wrangler devserving the production build, not just source:og:imageandsummary_large_image; all three h1s carry the brand; the JSON-LD graph resolves to one entity withsameAs./readme: 6,497 extractable words, 6 tables, 6<details>, 41 code blocks, 43 heading ids, 3 relative links rewritten, zero broken ones./compatibility: seeded D1 and confirmed real grouped<table>rows with test paths as text outside the SVG.<title>and no contradicting robots directive.lastmodvalues sourced from D1.vp checkpasses (3,113 files formatted, 1,322 linted/typechecked).knipexits 0. Full unit suite: 11 failures on this branch vs 15 on cleanmain— I stashed and re-ran to confirm none are new. The remainder are pre-existingpackages/vinextinternals (app-route-graph,app-ssr-stream,file-matcher,oxlint-prefer-shared-utils); the build-heavy ones pass in isolation and flake under parallel load.No changeset: nothing under
packages/*is touched, socreate-changeset.mtsgenerates nothing. This is unlike #2989, which needed suppressing because it edited the four package manifests.Not in this PR
Worth more than everything above combined, and none of it is code:
blog.cloudflare.com/vinext/(and the/ja-jp/and/ko-kr/localisations), and swap thebenchmarks.vinext.workers.devlink forvinext.dev/benchmarks.developers.cloudflare.com/workers/framework-guides/web-apps/nextjs/, which does not mention it at all.@vinext/cloudflare,create-vinext-appand@vinext/types— all four manifests sethomepage, but onlyvinexthas published since, so the other three still show no homepage on npmjs.com. Repo topics are also empty at 8,645 stars.