Repository navigation
fix(discovery): stop a //path sitemap entry from aborting the capture - #602
Conversation
… host A sitemap entry such as `http://example.test//alt` passed the host check, then `acceptEntry` rebuilt it with `new URL(pathname + search, baseOrigin)`. A pathname that starts with `//` is a scheme-relative reference for the URL parser, so the accepted route became `https://alt/`. Capture then threw `SameOriginViolation` and the whole run aborted after discovery. Clone the entry origin and assign `pathname` and `search` instead. The setters cannot change the host, and every other entry is rewritten exactly as before (checked old vs new on 658 real and edge-case entries: only the `//` and `///` cases differ). Every adapter that uses `fetchSitemap` goes through this one function. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
After the host fix, a sitemap entry `http://example.test//x` becomes the route `https://example.test//x`. Export maps `//x` and `/x/` to the same file (`x/index.html`), so a site that also links `/x/` captured every page and then failed at export with "Captured routes resolve to the same website path". Relative links on a page fetched at `//x` also resolve to `//…`. Collapse runs of `/` to one in sitemap entries and in page links. Both go through the two discovery primitives (`acceptEntry`, `resolvePageLink`), so every adapter that uses them gets the same route identity as export. Scheme-relative links (`//other.test/x`) are still dropped as off-origin. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A sitemap that lists both `/x/` and `/x` (or `//x`, now collapsed to `/x`) handed capture two routes that export writes to the same file, so the run captured every page and then failed with "Captured routes resolve to the same website path". Dedupe sitemap entries, and the thin-sitemap homepage link supplement, by the same document identity export uses (`normalizedUrl`: no fragment, no trailing slash, query kept). The first listed form wins. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Review finding: rebuilding the accepted origin with pathname/search setters fixes the host-escape bug correctly, but collapseRepeatedSlashes also rewrites the URL actually fetched. Repeated slashes are not generically equivalent source routes: a bounded local HTTP reproduction served distinct documents at /a//b and /a/b, and fetch preserved both paths. Current tests assert collapse without proving source redirect/canonical equivalence, so this can capture the wrong document while reporting success. Please preserve the accepted source pathname and use observed redirect/canonical evidence for deduplication; an unrepresentable collision should remain explicit. Add a distinct-route browser/HTTP regression alongside the repaired scheme-relative-host case. AI assistance: OpenAI gpt-6.1-sol via OpenCode reviewed source and reproduced the route-identity boundary. |
|
The route-preservation correction is now #611 (b33a9a5). Its real HTTP regression failed on merged main: four discovered URLs instead of six, silently omitting the distinct double-slash documents. After correction all original paths fetch their own server responses, the original same-origin setter repair remains, and an unverified portable collision remains an explicit error. 39 focused checks plus TypeScript and installed/relocated package workflows pass; fresh full hosted CI is running. AI assistance: OpenAI gpt-6.1-sol via OpenCode implemented and verified the follow-up in an isolated tracker-linked worktree. |
2 participants
What
Three small changes in
src/lib/extraction/sitemap.ts:acceptEntryrebuilt an accepted sitemap entry withnew URL(pathname + search, baseOrigin). A path that starts with//is read as scheme-relative there, sohttp://example.test//altbecamehttps://alt/. It now clones the entry origin and setspathnameandsearch, so the host cannot change.resolvePageLink). Export writes//x,/xand/x/to the same file (x/index.html). Keeping them apart only moves the failure to export: "Captured routes resolve to the same website path".normalizedUrl). A sitemap that lists/x/and/x(or//x) now gives one route. The first listed form wins.Why
A production import failed twice today after 23 s with "Same-origin violation expected https://<site>, got: https://ru/". The site's sitemap has one entry
http://<site>//ru, and the server serves it. One bad sitemap path should not stop a 650-page capture. Fixing only the host would make the same site fail later at export, after capturing every page.The same-origin guard in capture is unchanged. It is the safety boundary. The bug was that discovery made an off-site URL from an on-site entry.
How to test
npx vitest run src/lib/extraction/sitemap.test.ts. The new test for//paths fails onmain(routes come back ashttps://alt/,https://deep/) and passes here./x/and//x. Onmain, liberate stops in seconds withSameOriginViolation. With only change 1, it captures every page and then fails at export. With all three changes it exits 0 and writesx/index.htmlonce.🤖 Generated with Claude Code