feat(source-facts): name the unclosed fence instead of failing silently (closes #82)

docs/adr/0008-an-unclosed-fence-is-reported-not-guessed-shut.md

@@ -0,0 +1,78 @@
+---
+status: accepted
+---
+
+# An unclosed fence is reported, not guessed shut
+
+`source_facts/markdown.py` tracks fenced code blocks so that a JSON sample inside one never
+becomes a source fact. Following CommonMark, a closing fence must carry no info string: a line
+reading ```` ```json ```` opens a fence, it never closes one.
+
+Some sources close their fences that way anyway — pairing ```` ```json ```` with ```` ```json ````
+reads naturally to a human, and a renderer that is lenient about it will display the document
+correctly. Under the strict rule the scan treats everything after that line as fence content, so
+the rest of the document is never read: zero facts, no error, and a result identical to a source
+that genuinely has no structure.
+
+The strict rule stays. The alternative — accepting an info-string line as a close — is a guess
+about which of two readings the author meant, and it is wrong in the case that costs the most: two
+adjacent opening fences (a JSON request sample followed by a JSON response sample, each opened and
+closed in the ordinary way, with a stray info string on one close) would be read as one open and
+one close, putting the sample *between* them outside any fence. Its contents then become source
+facts. A fabricated fact blocks a correct extraction under the fail-closed completeness gate,
+which is the harm this project consistently refuses to risk (ADR 0007).
+
+What changes is that the failure is no longer silent. `SourceFacts` records the line where such a
+fence opened, the coverage projection carries that line, and the `SOURCE_FACTS_UNSCANNED` warning
+(ADR 0007) names it: the operator is told which line to open instead of being handed three possible
+causes to choose between. `verify-extraction` forecasts the same line before a run directory exists.
+
+The record is made only when the scan saw a line that *looks* like a close and was refused — an
+info string on the closing fence, or a marker that does not match the opening one. A fence that
+simply runs to the end of the document without any such line is not reported: CommonMark closes an
+unterminated fence at end of input, so every reader agrees with the scanner and nothing was lost by
+the strict rule. Reporting it anyway would send an operator to fix a source that is not broken, and
+the fact count after the "fix" would be unchanged.
+
+The warning also fires when the source is only *partly* unread — facts before the fence matched the
+extraction while everything after it went unread. That case is more dangerous than a wholly
+unscanned source, not less: the gate ran, found nothing wrong with the part it could see, and the
+report looks clean.
+
+## Considered options
+
+- Accepting an info-string line as a closing fence fixes the documents that pair their fences that
+  way, but it is a guess, and the case it gets wrong leaks a code sample into the fact inventory.
+  A missed fact costs a check that did not run; a fabricated one costs an operator who cannot ship
+  correct work.
+- Re-scanning leniently *only when* the strict scan ends inside a fence would bound the guess to
+  documents the strict scan definitely failed on. It is tempting and still rejected: a source that
+  is genuinely truncated mid-sample ends inside a fence too, and that is precisely when a lenient
+  re-scan reads the truncated sample as prose.
+- Failing the run on an unclosed fence would guarantee nobody ships an unread source, but a fence
+  that never closes is a defect in the *source*, and the pipeline's answer to a defective source is
+  to report it, not to refuse to produce the artifacts an operator needs in order to judge it.
+- Leaving the limit undisclosed — the state before this decision — makes an unread document
+  indistinguishable from an unstructured one, which is the exact confusion ADR 0007 exists to
+  remove.
+
+## Consequences
+
+A source whose fences are mismatched still yields nothing after that line, and the operator has to
+fix the source (or its acquisition path) before the gate can judge it. That is the accepted cost.
+
+The two Markdown scanners now differ on this point deliberately: `markdown_drafts/markdown.py`
+closes a fence on any line starting with the marker, info string or not, because its output is
+non-authoritative draft material that a human reviews, and a leaked sample there costs a reviewer
+one glance. `source_facts/markdown.py` feeds a fail-closed gate, where the same leak costs a
+correct extraction. The divergence is catalogued in the scanner-divergence follow-up rather than
+resolved by making one match the other.
+
+No benchmark source currently trips this: a scan across all thirteen cases found zero
+info-string closes and zero documents ending inside a fence. This decision is therefore about a
+failure mode that is cheap to disclose and expensive to misread, not about a fire being put out.
+
+**Falsified if:** a refused closing fence stops being reported, or the scan starts guessing fences
+shut. Concretely, this decision no longer holds when `loop_apidoc/source_facts/markdown.py` treats a
+line carrying an info string as a closing fence, or when `loop_apidoc/validate/fact_coverage.py`
+stops naming the line where the unclosed fence opened.

docs/operator-manual.en.html

@@ -260,6 +260,7 @@ <h3 id="verify-extraction"><code>verify-extraction</code> — check that the ext
 <p>The <strong>source-fact gate</strong> mechanically scans the manifest's Markdown sources for endpoint declarations, parameter tables and fenced example blocks, then matches them to the extraction by <code>(METHOD, path)</code>. When a matched source section documents fields or examples that the extraction dropped, the run fails closed — a silent omission is not the same as "the source does not say so." Naming the field in <code>missing</code> satisfies the gate, so it only ever forces a source-grounded gap, never an invention. Field names are resolved through <code>schema_ref</code> into <code>inventory.schemas</code> transitively, so factoring a shared request body out into a common type counts as deduplication, not an omission.</p>
 <p>A companion check rejects placeholder answers that defer the work, and it does so in two layers to avoid false positives. Phrases that explicitly name the extraction itself ("further extraction", "not yet extracted", 「需進一步擷取」) count anywhere in a value, since a real API description never discusses its own extraction. Generic placeholders ("TBD", "to be determined", 「待補」) count only when they are the <em>entire</em> field, because "amount to be determined at capture" is legitimate API prose. ASCII phrases match on word boundaries — CJK has none, so those stay substring matches. Without this check a run could finish as <em>passed</em> with empty artifacts.</p>
 <p><strong>Know the scope limit.</strong> The scan only recognises well-structured Markdown: headings, GFM tables with a separator row, and fenced code blocks. A source flattened into long single lines — an HTML-to-text dump, for instance — yields zero facts, and the gate is a no-op on it. So <em>a clean gate exit is not by itself evidence of a complete extraction</em>; it only proves nothing contradicted the facts that could be mechanically read. On unstructured sources, keep relying on <code>review.html</code> and human review.</p>
+<p><strong>When the message names a line, the cause is already settled.</strong> A fence opened there and a later line that looks like its close was refused — most often a closing fence carrying an info string, such as one ending with <code>```json</code>, which CommonMark reads as opening a new fence rather than closing the old one — so everything after that line went unread. It is reported even when facts before the fence matched the extraction: a source that goes unread halfway through is more dangerous than one never read at all, because the gate appears to have worked. Fix the source and re-extract; the extraction itself needs no change. A fence that simply runs to the end of the document with no refused close is not reported — CommonMark closes an unterminated fence at end of input, so nothing was lost. The reasoning is recorded in <code>docs/adr/0008-an-unclosed-fence-is-reported-not-guessed-shut.md</code>.</p>
 <p><strong>That limit is no longer silent.</strong> Every <code>assemble</code> records, per manifest source, how many facts were scanned and how many of them matched an extracted endpoint identity, and reports both failure shapes as warning-severity <code>SOURCE_FACTS_UNSCANNED</code> validation issues: <strong>zero facts</strong> (no endpoint facts were scanned from that source — open it first: content flattened into single lines, or an unconverted PDF/Word file, calls for re-running preprocessing along a table-preserving path such as <code>normalize-html-snapshot</code> or <code>preprocess</code>, and re-reading it achieves nothing; a structurally sound source whose endpoints are not written as <code>METHOD /path</code> — a bare URL with the method stated in prose, or a null-path webhook — will keep the warning permanently, because the scanner does not infer a missing method (ADR 0007), and the extraction may be entirely correct; a prose-only source legitimately lands here) and <strong>zero matches</strong> (facts were scanned but none matched the extraction by <code>METHOD /path</code> — check whether the extraction missed the endpoints that source documents). The severity is always warning and never blocks a run: a legitimate prose-only source with no parameter tables lands in the zero-fact class, and failing it would read "could not be measured" as "is wrong". It does count against the documentation-quality score under source grounding, so two runs differ in score when one had more sources the gate never judged. <code>verify-extraction</code> forecasts the same thing on stderr, before you pay for plan→generate; the forecast stays out of <code>--json</code> and never changes the exit code. The reasoning is recorded in <code>docs/adr/0007-source-fact-scanning-stays-limited-to-well-structured-markdown.md</code>.</p>
 <h3 id="focus-directives"><code>--focus</code> — task-specific extraction focus directives</h3>
 <pre><code class="language-bash">uv run loop-apidoc verify-extraction --sources ./sources --extraction ./work --focus ./focus.json

docs/operator-manual.html

@@ -259,6 +259,7 @@ <h3 id="verify-extraction"><code>verify-extraction</code> — 檢查擷取 JSON
 <p><strong>來源事實閘</strong>會機械掃描 manifest 中的 Markdown 來源,取出端點宣告、參數表與圍籬範例區塊,再以 <code>(METHOD, path)</code> 與擷取結果對照。一旦對上的來源小節寫了欄位或範例、擷取卻交回空的,就 fail closed——靜默遺漏不等於「來源沒寫」。要主張來源沒寫,在 <code>missing</code> 裡具名該欄位即可通過,所以這道閘只會逼出有據可查的缺口,不會逼出捏造。欄位名會沿 <code>schema_ref</code> 遞迴解析進 <code>inventory.schemas</code>,因此把共用 request body 抽成共用型別算去重複,不算遺漏。</p>
 <p>另一道檢查攔下佔位式延後答案,並分兩層以避免誤判。明確指涉「擷取這件事」的說法(<em>further extraction</em>、<em>not yet extracted</em>、「需進一步擷取」)出現在值的任何位置都算,因為真實 API 描述不會提到自己的擷取流程;泛用佔位字(<em>TBD</em>、<em>to be determined</em>、「待補」)則只有在<strong>整個欄位就只有這句</strong>時才算,因為「amount to be determined at capture」是合法的 API 描述。英文片語以詞界比對,CJK 沒有詞界可言,維持子字串比對。少了這道檢查,run 會以 <em>passed</em> 收場而產物是空的。</p>
 <p><strong>請注意適用範圍。</strong>這道掃描只認得結構良好的 Markdown:標題、含分隔列的 GFM 表格、圍籬程式碼區塊。若來源被壓成一行行超長文字(例如 HTML 轉純文字的傾印檔),掃描結果為零筆事實,這道閘對它就完全沒有作用。因此<em>閘門乾淨通過本身並不等於擷取完整</em>,它只證明「機械讀得到的事實」沒有被違反。面對非結構化來源,仍要靠 <code>review.html</code> 與人工核對。</p>
+<p><strong>訊息點名行號時,成因已經確定。</strong>那一行開啟的圍籬其後出現過一個「長得像關閉、卻不算關閉」的行(最常見的是關閉行帶了 info string,例如以 <code>```json</code> 結尾——依 CommonMark 那不算關閉,而是又開了一個新圍籬),因此該行之後的內容完全沒有被讀到。即使前半有事實對上 extraction 也照樣回報:讀到一半才失效比全篇未讀更危險,因為閘門看起來運作正常。修好來源再重新擷取即可,不必動 extraction。圍籬單純沒有收尾、其後沒有疑似關閉行的文件不在此列——CommonMark 在檔尾關閉未終止的圍籬,那種文件沒有任何內容因此遺失。理由記在 <code>docs/adr/0008-an-unclosed-fence-is-reported-not-guessed-shut.md</code>。</p>
 <p><strong>這個限制不再是靜默的。</strong>每次 <code>assemble</code> 都會逐份 manifest 來源記下「掃出幾筆事實、其中幾筆對得上擷取的端點識別」,並把兩種失能寫成 warning 級的 <code>SOURCE_FACTS_UNSCANNED</code> 驗證問題:<strong>零事實</strong>(這份來源掃不出任何端點事實。先看它屬於哪一種:內容被壓平成單行、或未轉換的 PDF/Word,補救方向是改走保留表格結構的前處理路徑,例如 <code>normalize-html-snapshot</code> 或 <code>preprocess</code>,重讀來源沒有用;結構完好但端點沒寫成 <code>METHOD /path</code>(只給完整 URL、method 寫在散文裡,或本來就是 path 為 null 的 webhook),掃描器不會去推測缺少的 method——那是 ADR 0007 拒絕的推論,因此這筆警告會長期存在,擷取本身可能完全正確;純散文來源則本來就會落在這裡)與<strong>零匹配</strong>(掃出了事實,但沒有一筆能以 <code>METHOD /path</code> 對上擷取,補救方向是檢查擷取是否漏了這份來源記載的端點)。severity 恆為 warning、不阻擋 run——純散文、本來就沒有參數表的合法來源會落在零事實這一類,擋下它等於把「量不到」誤判成「錯」——但會計入文件品質分數的 source grounding 類別,讓兩次 run 的分差能表達「這次有更多來源沒被檢查」。<code>verify-extraction</code> 會在 stderr 預告同一件事,讓你在付出 plan→generate 成本之前就能改用別的前處理指令;預告不進 <code>--json</code>、不改退出碼。理由記在 <code>docs/adr/0007-source-fact-scanning-stays-limited-to-well-structured-markdown.md</code>。</p>
 <h3 id="focus-directives"><code>--focus</code> — 依任務對擷取下重點指令</h3>
 <pre><code class="language-bash">uv run loop-apidoc verify-extraction --sources ./sources --extraction ./work --focus ./focus.json

loop_apidoc/agentcli/fact_coverage.py

@@ -40,5 +40,10 @@ def build_fact_coverage(
             in identities
         )
         coverage[source.relative_path] = FactCoverage(
-            facts=len(endpoints), matched=matched)
+            facts=len(endpoints),
+            matched=matched,
+            unclosed_fence_line=(
+                entry.unclosed_fence_line if entry is not None else None
+            ),
+        )
     return coverage

loop_apidoc/agentcli/verify.py

@@ -108,11 +108,19 @@ def verify_extraction(
 
 def _forecast(coverage) -> list[str]:
     """把投影寫成人可讀的一行一份來源。"""
-    return [
-        f"{source}:掃出 0 筆端點事實" if entry.facts == 0
-        else f"{source}:{entry.facts} 筆事實無一對上 extraction 的端點"
-        for source, entry in unscanned_sources(coverage)
-    ]
+    lines: list[str] = []
+    for source, entry in unscanned_sources(coverage):
+        if entry.unclosed_fence_line is not None:
+            lines.append(
+                f"{source}:第 {entry.unclosed_fence_line} 行的圍籬未關閉,"
+                "其後的內容全部沒被讀到"
+            )
+        elif entry.facts == 0:
+            lines.append(f"{source}:掃出 0 筆端點事實")
+        else:
+            lines.append(
+                f"{source}:{entry.facts} 筆事實無一對上 extraction 的端點")
+    return lines
 
 
 def preview_falsified_expectations(

loop_apidoc/source_facts/markdown.py

@@ -90,6 +90,8 @@ def scan_markdown(relative_path: str, text: str) -> SourceFacts:
                 fence.group("marker"), fence.group("info")
             ):
                 state.close_fence()
+            elif fence:
+                state.reject_close()
             continue
         if fence:
             state.flush_table()
@@ -135,6 +137,9 @@ def scan_markdown(relative_path: str, text: str) -> SourceFacts:
         relative_path=relative_path,
         endpoints=state.endpoints,
         error_codes=state.error_codes,
+        # 掃完仍在圍籬內 ⇒ 這份文件從那一行起沒有被讀過。判定維持嚴格(見 ADR 0008),
+        # 但失效不再是靜默的。
+        unclosed_fence_line=state.fence_line if state.rejected_close else None,
     )
 
 
@@ -152,6 +157,9 @@ def __init__(self, relative_path: str) -> None:
         self.declaring_level = 0
         self.fence_marker: str | None = None
         self.fence_length = 0
+        self.fence_line: int | None = None
+        # 目前這道圍籬內,是否出現過「長得像關閉、卻不算關閉」的行。
+        self.rejected_close = False
         self.table: list[tuple[int, str]] = []
         self.previous = ""
         # 錯誤碼表不依附端點,所以索引與累積都在來源層級。
@@ -176,6 +184,8 @@ def open_fence(
                 )
         self.fence_marker = marker[0]
         self.fence_length = len(marker)
+        self.fence_line = index
+        self.rejected_close = False
         self.previous = ""
 
     def closes_fence(self, marker: str, info: str) -> bool:
@@ -185,9 +195,14 @@ def closes_fence(self, marker: str, info: str) -> bool:
             and len(marker) >= self.fence_length
         )
 
+    def reject_close(self) -> None:
+        self.rejected_close = True
+
     def close_fence(self) -> None:
         self.fence_marker = None
         self.fence_length = 0
+        self.fence_line = None
+        self.rejected_close = False
         self.previous = ""
 
     def flush_table(self) -> None:

loop_apidoc/source_facts/models.py

@@ -80,6 +80,12 @@ class SourceFacts(BaseModel):
     endpoints: list[EndpointFact] = Field(default_factory=list)
     #: 這份來源以表格結構明確記載的錯誤碼,依出現順序。
     error_codes: list[ErrorCodeFact] = Field(default_factory=list)
+    #: 開在第幾行、其後出現過「長得像關閉、卻不算關閉」的行、且掃到檔尾仍未關閉
+    #: 的那道圍籬(沒有就是 None)。從該行起整份文件都被當成在圍籬內,掃出零筆且
+    #: 毫無錯誤——這個欄位存在的唯一目的,是讓那次靜默失效在報告裡有具名的成因。
+    #: 圍籬單純沒有收尾(其後沒有任何疑似關閉行)不算:CommonMark 在檔尾關閉未
+    #: 終止的圍籬,那份文件在每個讀者眼中都一樣,沒有任何內容因為我們的判定而遺失。
+    unclosed_fence_line: int | None = None
 
 
 class FactIndex(BaseModel):