docs(source-facts): pin the two scanners' divergences instead of unifying them (closes #83)

AGENTS.md

@@ -110,7 +110,7 @@ The CLI commands include source acquisition, quality, assembly, analysis, and Fo
 | `loop_apidoc/url_catalog.py` | reproducible URL navigation catalog behind `catalog-url`/`select-url`: `fetch_catalog` (bounded HTTP GET of *one* entry page, size-capped, `CatalogFetchError`), `build_catalog` (parse only sidebar/nav lists into `CatalogNode`s — entry-page fragments kept as `anchor` section identities; links are recorded, never followed), `select_catalog` (pure filter by branch/term/URL; widens nothing, fetches nothing) |
 | `loop_apidoc/url_corpus.py` | token-efficient cached URL corpus behind `cache-url-pages`/`cache-url-entry`/`related-url-pages`: `cache_catalog_pages` (fetch each catalog URL once — anchors of the same document become `sections` — and write content-addressed `raw/<sha256>.html` + `body/<sha256>.txt`; failures become `status: fetch_failed` entries, not exceptions; an un-rendered SPA shell probes only same-origin `/swagger.json`, `/openapi.json`, `/v3/api-docs`, and `/api-doc/v3/sections`; only JSON with an `openapi` or `swagger` root field is stored as a separate corpus source, while failed/non-spec/generic-JSON responses are silently not recorded and the CLI warns of the shell count on stderr), `extract_page_metadata` (pure: title/headings/body/internal links/`action:`+error-code entities from `<main>`), `find_related_pages` (pure evidence-based scoring — same branch / in-out links / shared entities — returning candidate cards without loading body text) |
 | `loop_apidoc/gitbook_llms.py` | deterministic GitBook `llms.txt` filtering/cache with safe path preservation, URL sidecars, and coverage |
-| `loop_apidoc/markdown_drafts/` | separate non-authoritative, line-cited Markdown endpoint/table/example drafts; never alters `source_facts` validation |
+| `loop_apidoc/markdown_drafts/` | separate non-authoritative, line-cited Markdown endpoint/table/example drafts; never alters `source_facts` validation. Its scanner deliberately diverges from `source_facts/markdown.py` — each is looser where its own job needs it, neither is uniformly stricter, and they must not be unified, since a change made for draft output would silently move a fail-closed gate (ADR 0009, pinned in `tests/source_facts/test_scanner_divergence.py`) |
 | `loop_apidoc/extraction_scaffold/` | pure projection of Markdown drafts into review-only extraction-shaped inventory/endpoint JSON; `write.py` is this feature's sole atomic output exit, and agents must copy/review output before it is used as real extraction |
 | `loop_apidoc/html_snapshot.py` | `normalize-html-snapshot`: `html_to_markdown` (pure: readable main-document text, no invented content) + `normalize_html_snapshot` (writes the Markdown and a `.source.json` sidecar binding it to the raw file's URL + sha256) |
 | `loop_apidoc/rendered_url.py` | first-class offline browser-rendered URL import: validates original/canonical URL, timezone-aware capture metadata, capture method, safe immutable destinations, and SHA-256; writes the unchanged HTML/Markdown source, versioned provenance sidecar, and `fetched_rendered` coverage. Its read-side verifier binds coverage → sidecar → manifest local source and fails closed on any mismatch before an origin probe. |
@@ -122,7 +122,7 @@ The CLI commands include source acquisition, quality, assembly, analysis, and Fo
 | `loop_apidoc/docx_normalization.py` + `docx_{models,validation,render,publish}.py` | stable DOCX facade plus bounded, fail-closed OOXML validation, deterministic rendering, and staged Markdown/`.source.json` publication with rollback on reported write failures; package validation scans every Word XML part for active DDE fields, markup alternatives, merged cells, and external content without executing or resolving relationships |
 | `loop_apidoc/adapters/fragments.py` | read-side I/O exit that materializes exact page/line/section/table-cell/JSON Pointer fragments from source artifacts; fragment digests use normalized fragment content, not whole-document bytes |
 | `loop_apidoc/shadow/` | opt-in legacy/Core compatibility sidecar: `models.py` (mode, diagnostics, comparison, summaries), `bridge.py` (pure manifest/plan → evidence/support proposals/metadata; a v1 exact reference owns its declared claim path while filename-only legacy citations degrade to `insufficient`/unverified), `runner.py` (in-memory deterministic verification and evidence-aware projections through validate only), `report.py` (successful `core/*.json` plus `core/projections/`, or safe `core/error.json`; this package's only file-I/O exit) |
-| `loop_apidoc/source_facts/` | deterministic source-fact inventory feeding the semantic completeness gate (issue #14): `models.py` (`EndpointFact`/`SourceFacts`/`FactIndex`, `by_identity()` keeping only the **intersection** when several sources document one `(METHOD, path)` — an overview index table or a deprecated v1 section would otherwise widen the requirement past what the extraction was right to ignore; ambiguity fails open), `markdown.py` (`scan_markdown`, pure: endpoint declarations, parameter-table field names — only tables whose first header cell is name-like, with nested-row decoration stripped and group-label rows skipped — and fenced example-block counts, fence-aware so code samples never leak facts; a closing fence must carry no info string (CommonMark), and when the scan ends inside a fence `SourceFacts.unclosed_fence_line` records where it opened so the `SOURCE_FACTS_UNSCANNED` warning names the line instead of listing possible causes — the strict rule is never relaxed into guessing a fence shut, ADR 0008. **Scope limit:** only well-structured Markdown yields facts; a flattened HTML-to-text dump yields none and the gate is a no-op on it — an accepted trade-off, since guessing structure would manufacture false facts and a false fact blocks a correct extraction — the cost is disclosed at runtime as `SOURCE_FACTS_UNSCANNED`, and `collect.py` is named in ADR 0007's falsification condition), `collect.py` (`collect_facts`, the package's only read: manifest-named Markdown sources → `FactIndex`; unreadable sources are skipped, since manifest coverage already reports them), `gate.py` (`source_fact_violations`, pure: for every extracted endpoint that matches a fact by `(METHOD, path)`, a documented field absent from every structural position — and not named in `missing[]` — or a documented example with an empty `examples[]` is a violation; no match ⇒ no judgement), `deferral.py` (`deferral_violations`, pure: rejects placeholder answers like "requires further extraction"/「需進一步擷取」 outside `missing[]`). `markdown.py` also recognises **error-code tables** into per-source `ErrorCodeFact`s, and `FactIndex.documented_error_codes()` unions them across sources into the documented error-code floor — union, not the `by_identity()` intersection, because different documents tabulate different code sets rather than competing accounts of one thing (ADR 0005). Recognition is strict: an unambiguous code header, or a generic one (`代碼`/`code`/`status code`) corroborated by an enclosing error section; at least two columns; and one malformed data row discards the whole table rather than lowering the floor silently. **`gate.py` deliberately does not consume the floor** — only focus directives are judged against it. Error codes are a document-level shared catalogue with no endpoint to match against, so an unscoped requirement would hit integrations that correctly implement only part of a provider; requiring exhaustiveness stays something a requester asks for (ADR 0006). That is a decision, not an oversight, and `source_facts/gate.py` is named in ADR 0006's falsification condition |
+| `loop_apidoc/source_facts/` | deterministic source-fact inventory feeding the semantic completeness gate (issue #14): `models.py` (`EndpointFact`/`SourceFacts`/`FactIndex`, `by_identity()` keeping only the **intersection** when several sources document one `(METHOD, path)` — an overview index table or a deprecated v1 section would otherwise widen the requirement past what the extraction was right to ignore; ambiguity fails open), `markdown.py` (`scan_markdown`, pure, and deliberately not shared with the draft scanner — ADR 0009: endpoint declarations, parameter-table field names — only tables whose first header cell is name-like, with nested-row decoration stripped and group-label rows skipped — and fenced example-block counts, fence-aware so code samples never leak facts; a closing fence must carry no info string (CommonMark), and when the scan ends inside a fence `SourceFacts.unclosed_fence_line` records where it opened so the `SOURCE_FACTS_UNSCANNED` warning names the line instead of listing possible causes — the strict rule is never relaxed into guessing a fence shut, ADR 0008. **Scope limit:** only well-structured Markdown yields facts; a flattened HTML-to-text dump yields none and the gate is a no-op on it — an accepted trade-off, since guessing structure would manufacture false facts and a false fact blocks a correct extraction — the cost is disclosed at runtime as `SOURCE_FACTS_UNSCANNED`, and `collect.py` is named in ADR 0007's falsification condition), `collect.py` (`collect_facts`, the package's only read: manifest-named Markdown sources → `FactIndex`; unreadable sources are skipped, since manifest coverage already reports them), `gate.py` (`source_fact_violations`, pure: for every extracted endpoint that matches a fact by `(METHOD, path)`, a documented field absent from every structural position — and not named in `missing[]` — or a documented example with an empty `examples[]` is a violation; no match ⇒ no judgement), `deferral.py` (`deferral_violations`, pure: rejects placeholder answers like "requires further extraction"/「需進一步擷取」 outside `missing[]`). `markdown.py` also recognises **error-code tables** into per-source `ErrorCodeFact`s, and `FactIndex.documented_error_codes()` unions them across sources into the documented error-code floor — union, not the `by_identity()` intersection, because different documents tabulate different code sets rather than competing accounts of one thing (ADR 0005). Recognition is strict: an unambiguous code header, or a generic one (`代碼`/`code`/`status code`) corroborated by an enclosing error section; at least two columns; and one malformed data row discards the whole table rather than lowering the floor silently. **`gate.py` deliberately does not consume the floor** — only focus directives are judged against it. Error codes are a document-level shared catalogue with no endpoint to match against, so an unscoped requirement would hit integrations that correctly implement only part of a provider; requiring exhaustiveness stays something a requester asks for (ADR 0006). That is a decision, not an oversight, and `source_facts/gate.py` is named in ADR 0006's falsification condition |
 | `loop_apidoc/focus/` | requester-authored extraction focus directives: `models.py` (strict `extra="forbid"` `FocusDirective`/`FocusResponse` contracts — `kind` is the sole determinant of severity, `intent` the sole determinant of anchor type, and the only two outcomes are `satisfied`/`not_found`; there is deliberately no "not applicable", since whether a directive applies is the requester's judgement), `loader.py` (this package's only read exit: parses `focus.json` and `<extraction>/focus-response.json`, `FocusInputError`), `gate.py` (pure: directive↔response correspondence, intent↔anchor-type agreement, anchor resolution against the extraction, and the requirement that a `not_found` answer account for every readable manifest source; also projects anchor evidence for the shared exact-evidence verifier rather than verifying it separately), `fields.py`/`codes.py` (pure anchor vocabularies built on the shared field-name and typed error-catalogue readers), `report.py` (this package's only write exit: `<run-dir>/focus/focus-report.{json,zh-TW.md}`). Structural violations fold into `agentcli/gate.py`, so they fail before a run directory exists. The documented error-code floor is judged outside this package, in `validate/focus.py` (`omitted_error_codes`, shared by the `assemble` issue and the `verify-extraction` forecast so the two cannot disagree); `codes.py` stays the anchor vocabulary that resolves a reported code against the typed catalogue and is deliberately not the floor's source. Focus material never reaches provenance, the score, or Foundry (ADR 0004). |
 | `loop_apidoc/agentcli/identity.py` | the one definition of an endpoint's cross-file identity key (`METHOD /path`, or `METHOD (webhook) <summary>` when a webhook's path is null), shared by the cross-file invariants and the focus anchor resolver |
 | `loop_apidoc/extraction/` | shared models + utilities (models, stages, questions, store, jsonblock) used by the agent extraction |

docs/adr/0009-the-two-markdown-scanners-stay-separate.md

@@ -0,0 +1,67 @@
+---
+status: accepted
+---
+
+# The two Markdown scanners stay separate, and their divergences are pinned
+
+Two modules read structured Markdown and pull endpoint declarations, parameter tables, and
+fenced examples out of it: `source_facts/markdown.py` and `markdown_drafts/markdown.py`. They
+duplicate the shapes they look for and disagree about several of them. Read cold, that looks
+like an oversight waiting to be DRYed up.
+
+It is not. They answer to different consequences. `source_facts` feeds the fail-closed semantic
+completeness gate: everything it reads becomes something the extraction *must* produce, so a fact
+it invents blocks a correct extraction. `markdown_drafts` produces non-authoritative draft JSON
+that an agent copies, re-reads against the source, and fills in by hand; a shape it misses or
+over-reads costs a reviewer one more glance. Strictness that is correct on one side is wrong on
+the other, in both directions.
+
+## The divergences, as measured
+
+Neither scanner is uniformly the stricter one — each is looser exactly where its own job needs
+it. `tests/source_facts/test_scanner_divergence.py` pins every row.
+
+| shape | `source_facts` | `markdown_drafts` | why |
+| --- | --- | --- | --- |
+| bare / backticked / list / bold declaration line | reads | ignores | The draft's unit is a section, and a declaration with no heading opens none. The gate has no sections: it reads a fact wherever the fact is. |
+| heading that *is* the declaration | reads | reads | The least ambiguous shape there is. |
+| heading with a prefix (`## 支付 GET /a`) | ignores | reads | The gate anchors its match at the start; the draft searches the whole line. Anchoring is what keeps "見 POST /pay 的說明" out of the fact inventory. |
+| lowercase method | ignores | reads (case-insensitive) | Same trade: case-insensitive matching widens the prose surface a fail-closed gate is exposed to. |
+| GitBook's `` `GET` `` + `` `/a` `` pair | ignores | reads | **A gap, not a decision** — see below. |
+| declaration inside prose | ignores | ignores | Reading it would be guessing, on either side. |
+| unlabelled parameter table | reads | ignores | The draft needs a `**Request**`-style label to know which section a field belongs to; the gate only asks whether the source documents the field at all. |
+| fenced block in any language | counts as an example | only whitelisted languages | The gate needs the fact "an example exists here"; the draft has to paste the example, so it takes only languages it can carry. |
+| closing fence carrying an info string | refuses (strict CommonMark) | accepts | ADR 0008. Leniency in the gate turns two adjacent samples into one open/close pair and leaks the material between them into the fact inventory. |
+
+## Considered options
+
+- Sharing one scanner, or one set of regexes, removes the duplication but couples a fail-closed
+  gate to a draft generator: a change made to improve draft output silently moves the gate, and
+  the symptom is a run that fails for a reason nobody edited. The duplication is the cheaper of
+  the two costs.
+- Converging the gate onto the draft's looser rules (prefix headings, lowercase methods) would
+  close real recogniser gaps, but it widens the prose surface of the side where a false positive
+  is expensive. Measured against the thirteen benchmark cases, all three candidate widenings —
+  prefix headings, lowercase methods, and the GitBook pair — occur **zero** times, so there is
+  currently no coverage to win and a fail-closed gate to risk.
+- Converging the draft onto the gate's stricter rules loses draft coverage for no benefit: the
+  draft is reviewed by a human before anything downstream trusts it.
+
+## Consequences
+
+The GitBook `` `GET` `` + `` `/a` `` pair is the one row that is a gap rather than a decision.
+It is a literal declaration, not an inference, and recognising it would not weaken the gate. It
+stays unrecognised only because no source in the current corpus uses it, and widening a
+fail-closed gate for a shape nobody has is speculative work with a real downside. A source that
+uses it is the trigger to close the gap — and, until then, such a source shows up as a
+`SOURCE_FACTS_UNSCANNED` warning rather than as silence (ADR 0007), which is what makes waiting
+safe.
+
+Anything that changes a row in the table changes this decision, and the pinned test is where that
+shows up first.
+
+**Falsified if:** the two scanners stop being separately governed. Concretely, this decision no
+longer holds when `loop_apidoc/source_facts/markdown.py` and
+`loop_apidoc/markdown_drafts/markdown.py` share a scanning implementation or a common regex
+module, or when `tests/source_facts/test_scanner_divergence.py` no longer pins both sides of each
+divergence.

loop_apidoc/markdown_drafts/markdown.py

@@ -1,4 +1,14 @@
-"""Pure, conservative scanning of structured Markdown API facts."""
+"""Pure, conservative scanning of structured Markdown API facts.
+
+This is the draft side: its output is non-authoritative material an agent copies,
+re-reads against the source, and fills in by hand. `source_facts/markdown.py` is the
+other scanner, and the two deliberately disagree — each is looser exactly where its own
+job needs it, and neither is uniformly the stricter one. Do not unify them: a change
+made to improve draft output would silently move a fail-closed gate. The divergence
+table and its reasoning live in
+`docs/adr/0009-the-two-markdown-scanners-stay-separate.md`, pinned row by row in
+`tests/source_facts/test_scanner_divergence.py`.
+"""
 
 from __future__ import annotations
 

loop_apidoc/source_facts/markdown.py

@@ -8,6 +8,11 @@
 掃得出事實。把 HTML 壓平成長單行的來源會掃出零個事實,閘門對它等同無效。
 這是接受的取捨,不是待修的 bug——要涵蓋那類來源,得先在取源階段把結構
 還原(見 `html_snapshot.py`),而不是在這裡猜。
+
+**與 `markdown_drafts/markdown.py` 的分歧是刻意的**,不是重複實作待統一:那一套餵的
+是給人審的非權威草稿,這一套餵的是 fail-closed 閘門,兩邊的寬鬆處各自朝自己的用途
+交錯放寬。逐條對照與理由見 `docs/adr/0009-the-two-markdown-scanners-stay-separate.md`,
+每一格由 `tests/source_facts/test_scanner_divergence.py` 釘住。
 """
 
 from __future__ import annotations