fix(html-snapshot): stop merged cells and nested tables misaligning parameter tables (closes #81)

AGENTS.md

@@ -112,7 +112,7 @@ The CLI commands include source acquisition, quality, assembly, analysis, and Fo
 | `loop_apidoc/gitbook_llms.py` | deterministic GitBook `llms.txt` filtering/cache with safe path preservation, URL sidecars, and coverage |
 | `loop_apidoc/markdown_drafts/` | separate non-authoritative, line-cited Markdown endpoint/table/example drafts; never alters `source_facts` validation |
 | `loop_apidoc/extraction_scaffold/` | pure projection of Markdown drafts into review-only extraction-shaped inventory/endpoint JSON; `write.py` is this feature's sole atomic output exit, and agents must copy/review output before it is used as real extraction |
-| `loop_apidoc/html_snapshot.py` | `normalize-html-snapshot`: `html_to_markdown` (pure: readable main-document text, no invented content) + `normalize_html_snapshot` (writes the Markdown and a `.source.json` sidecar binding it to the raw file's URL + sha256) |
+| `loop_apidoc/html_snapshot.py` | `normalize-html-snapshot`: `html_to_markdown` (pure: readable main-document text, no invented content; `colspan`/`rowspan` expand into a rectangular grid — the spanning cell's text stays in its own column and the covered columns are left blank so a group-title row stays distinguishable from a parameter row, while `rowspan` carries the text down its own column; out-of-range/non-numeric spans count as 1, overlapping spans discard the whole table, a multi-row `thead` merges into one header row, and a nested table renders as its own table instead of appending its rows to the enclosing one — a misaligned parameter table becomes a source fact nobody wrote) + `normalize_html_snapshot` (writes the Markdown and a `.source.json` sidecar binding it to the raw file's URL + sha256) |
 | `loop_apidoc/rendered_url.py` | first-class offline browser-rendered URL import: validates original/canonical URL, timezone-aware capture metadata, capture method, safe immutable destinations, and SHA-256; writes the unchanged HTML/Markdown source, versioned provenance sidecar, and `fetched_rendered` coverage. Its read-side verifier binds coverage → sidecar → manifest local source and fails closed on any mismatch before an origin probe. |
 | `loop_apidoc/source_risk/` | deterministic pre-agent gate behind `inspect-source-risk`: `models.py` (versioned `SourceRiskReport`, findings and coverage), `inspect.py` (bounded read of manifest-bound UTF-8 Markdown/HTML/OpenAPI JSON/YAML; fixed rules, a 1,000-entry report cap with fail-closed `SR-FINDINGS-TRUNCATED`, and stable `source_binding_digest`; PDF/Word, invalid UTF-8, over-`max_bytes`, and other unscannable pending sources are blockers), `loader.py` (fail-loud schema/ruleset/verdict/manifest/source-binding verification plus deterministic reinspection of current bytes), `report.py` (`source-risk-report.{json,zh-TW.md}`; findings never echo matched payloads). Exit 0/1/2 means pass/reject/input error; source bytes are never rewritten. |
 | `loop_apidoc/source_quality/` | pre-extraction source quality gate behind `assess-sources --source-risk`: `models.py` (`QualityObservation`/`QualityFinding`, `FindingSeverity`, verdict `pass`/`reject` + `SourceDiffReport`; report embeds a verified `SourceRiskReport`; blocker observations may carry explicit HTTP(S) `required_source_refs`), `loader.py` (read side: manifest, agent-written observations JSON, and a completed assessment dir — `SourceQualityInputError`), `assess.py` (`assess_source_quality`, pure: manifest usability + observations + source-risk audit → findings; any blocker ⇒ `reject`; rejected reports aggregate a bounded ordered/de-duplicated `required_source_refs` list without fetching it), `diff.py` (`build_source_diff`, pure manifest-vs-manifest added/removed/changed), `report.py` (`write_reports` → `source-quality-report.{json,zh-TW.md}` + `source-diff.{json,md}`). `assemble` requires `--source-quality`, re-loads the reports, rejects missing/stale/mismatched embedded audits and internally inconsistent reports (strict `extra="forbid"` models; `verdict` must be `reject` exactly when a blocker finding exists, and `required_source_refs` must equal the ordered de-duplicated union from those blockers), and copies a passing pair into the run-dir's `source-quality/`. |

docs/HTML_SNAPSHOT_NORMALIZATION_SPEC.md

@@ -45,6 +45,15 @@ policy changes in this work.
    their descendants.  Do not add heuristic content scoring or site-specific rules.
 3. Preserve the existing output for headings, paragraphs, table cells (including
    escaped pipes), and `pre`/`code` block line breaks.
+3a. Render each table on its own grid.  A table owns only its own rows: a nested
+   table becomes a separate table block after its parent, and its text is not pulled
+   into the enclosing cell.  Expand `colspan`/`rowspan` into a rectangular grid — the
+   spanning cell's text stays in its own column and the columns it covers are left
+   blank, while `rowspan` carries the text down its own column.  Do not repeat a
+   spanning cell across the columns it covers: a row whose remaining cells are blank
+   is how downstream fact extraction tells a group-title row from a parameter row.
+   Out-of-range or non-numeric spans count as 1, overlapping spans discard the whole
+   table, and a multi-row `thead` merges into the single header row GFM allows.
 4. Render nested unordered and ordered lists as Markdown list items with deterministic
    indentation.  Preserve each item's readable text once; group/empty wrapper nodes
    must not produce duplicate lines.  Identical sibling items remain distinct: source

loop_apidoc/html_snapshot.py

@@ -5,10 +5,170 @@
 import json
 from datetime import datetime, timezone
 from hashlib import sha256
+from collections.abc import Callable
 from pathlib import Path
 
 from loop_apidoc.url_catalog import _Element, _TreeParser, _walk
 
+#: How far a cell may credibly span. Real parameter tables stay well inside these;
+#: anything larger is broken markup, and expanding it verbatim invents a table shape
+#: nobody wrote.
+_MAX_COLSPAN = 16
+_MAX_ROWSPAN = 64
+
+
+def _span(cell: _Element, attribute: str, limit: int) -> int:
+    """A span the document actually states, or 1.
+
+    Out-of-range and non-numeric values fall back to 1 rather than being clamped to
+    the limit: `colspan="9999"` is a broken document, and honouring it as "the widest
+    we allow" would invent a shape nobody wrote.
+    """
+    try:
+        value = int(cell.attrs.get(attribute, "1"))
+    except ValueError:
+        return 1
+    return value if 1 <= value <= limit else 1
+
+
+def _own_rows(table: _Element) -> tuple[list[_Element], list[_Element]]:
+    """This table's own rows as (header, body), nested tables left to themselves.
+
+    A nested table's rows belong to it; collecting every descendant `tr` appends them
+    to the enclosing table, where they line up against the wrong columns — and a
+    misaligned parameter table becomes a source fact the source never stated.
+
+    Rows are also collected from inside a `tr`, because the parser implements no
+    implied end tags: a source that omits `</tr>` nests every following row inside the
+    first one, and refusing to descend would drop the whole table body.
+    """
+    head: list[_Element] = []
+    body: list[_Element] = []
+    foot: list[_Element] = []
+
+    def visit(node: _Element, *, section: list[_Element]) -> None:
+        for child in node.children:
+            if not isinstance(child, _Element) or child.tag == "table":
+                continue
+            if child.tag == "tr":
+                section.append(child)
+                visit(child, section=section)
+                continue
+            if child.tag == "thead":
+                visit(child, section=head)
+            elif child.tag == "tfoot":
+                visit(child, section=foot)
+            else:
+                visit(child, section=section)
+
+    visit(table, section=body)
+    return head, body + foot
+
+
+def _nested_tables(table: _Element) -> list[_Element]:
+    """The outermost tables inside this one; each renders itself recursively."""
+    found: list[_Element] = []
+
+    def visit(node: _Element) -> None:
+        for child in node.children:
+            if not isinstance(child, _Element):
+                continue
+            if child.tag == "table":
+                found.append(child)
+            else:
+                visit(child)
+
+    visit(table)
+    return found
+
+
+def _table_grid(
+    rows: list[_Element],
+    cell_text: Callable[[_Element], str],
+) -> list[list[str]]:
+    """Expand `colspan`/`rowspan` into a rectangular grid, or `[]` if it cannot.
+
+    A spanning cell's text stays at its own position and the columns it covers are
+    left blank. Repeating it across those columns would be the alignment fix that
+    manufactures facts: downstream, a row whose remaining cells are blank is how a
+    group-title row ("Header", "支付類") is told apart from a parameter row, and a
+    repeated title fills those cells with a value the source never stated.
+
+    `rowspan` is the exception and repeats down its own column: there the carried
+    text is a real value for each of the rows it covers, and dropping it would strip
+    the group a nested field belongs to.
+
+    Overlapping spans return `[]` so the caller renders nothing — silently letting a
+    later cell overwrite a carried one produces a table that looks fine and is wrong,
+    the same bias the error-code reader takes when one row is malformed.
+    """
+    cells: dict[tuple[int, int], str] = {}
+    owners: dict[tuple[int, int], tuple[int, int]] = {}
+    for index, row in enumerate(rows):
+        column = 0
+        for cell in row.children:
+            if not isinstance(cell, _Element) or cell.tag not in {"th", "td"}:
+                continue
+            while (index, column) in cells:
+                column += 1
+            text = cell_text(cell).replace("|", r"\|")
+            columns = _span(cell, "colspan", _MAX_COLSPAN)
+            down = min(_span(cell, "rowspan", _MAX_ROWSPAN), len(rows) - index)
+            for offset in range(down):
+                for shift in range(columns):
+                    position = (index + offset, column + shift)
+                    if owners.get(position, (index, column)) != (index, column):
+                        return []
+                    owners[position] = (index, column)
+                    cells[position] = text if shift == 0 else ""
+            column += columns
+    if not cells:
+        return []
+    width = max(column for _, column in cells) + 1
+    height = max(index for index, _ in cells) + 1
+    return [
+        [cells.get((index, column), "") for column in range(width)]
+        for index in range(height)
+    ]
+
+
+def _merge_header(rows: list[list[str]]) -> list[str]:
+    """Fold a multi-row `thead` into the single header row GFM allows.
+
+    Demoting the extra rows to the body instead would hand the fact scanner a
+    parameter row built out of column titles.
+    """
+    merged: list[str] = []
+    for column in range(len(rows[0])):
+        parts: list[str] = []
+        for row in rows:
+            value = row[column]
+            if value and value not in parts:
+                parts.append(value)
+        merged.append(" ".join(parts))
+    return merged
+
+
+def _render_table(table: _Element, cell_text: Callable[[_Element], str]) -> str:
+    head, body = _own_rows(table)
+    grid = _table_grid(head + body, cell_text)
+    blocks: list[str] = []
+    if grid:
+        split = max(len(head), 1)
+        header = _merge_header(grid[:split]) if split > 1 else grid[0]
+        lines = [
+            "| " + " | ".join(header) + " |",
+            "| " + " | ".join(["---"] * len(header)) + " |",
+        ]
+        lines += ["| " + " | ".join(row) + " |" for row in grid[split:]]
+        blocks.append("\n".join(lines))
+    blocks += [
+        rendered
+        for rendered in (_render_table(nested, cell_text) for nested in _nested_tables(table))
+        if rendered
+    ]
+    return "\n\n".join(blocks)
+
 
 def html_to_markdown(html: str) -> str:
     """Extract readable main-document text without inventing content."""
@@ -47,7 +207,12 @@ def visit(node: _Element | str) -> None:
         visit(item)
         return " ".join("".join(parts).split())
 
-    def inline_text(item: _Element, *, exclude_lists: bool = False) -> str:
+    def inline_text(
+        item: _Element,
+        *,
+        exclude_lists: bool = False,
+        exclude_tables: bool = False,
+    ) -> str:
         """Render readable inline content without resolving or inventing links."""
         parts: list[str] = []
 
@@ -57,6 +222,11 @@ def visit(node: _Element | str) -> None:
                 return
             if node.tag in ignored or (exclude_lists and node.tag in {"ul", "ol"}):
                 return
+            # A nested table is rendered as its own table, so pulling its text into
+            # the enclosing cell would state the same rows twice — once as a run-on
+            # sentence that no longer says which value belongs to which column.
+            if exclude_tables and node.tag == "table":
+                return
             if node.tag == "a":
                 label = plain_text(node)
                 href = node.attrs.get("href", "")
@@ -80,26 +250,7 @@ def raw_text(item: _Element) -> str:
         return "".join(parts)
 
     def render_table(table: _Element) -> str:
-        rows: list[list[str]] = []
-        for row in (e for e in _walk(table) if e.tag == "tr"):
-            cells = [
-                inline_text(cell).replace("|", r"\|")
-                for cell in row.children
-                if isinstance(cell, _Element) and cell.tag in {"th", "td"}
-            ]
-            if cells:
-                rows.append(cells)
-        if not rows:
-            return ""
-        width = max(len(row) for row in rows)
-        rows = [row + [""] * (width - len(row)) for row in rows]
-        header, *body = rows
-        out = [
-            "| " + " | ".join(header) + " |",
-            "| " + " | ".join(["---"] * width) + " |",
-        ]
-        out += ["| " + " | ".join(row) + " |" for row in body]
-        return "\n".join(out)
+        return _render_table(table, lambda cell: inline_text(cell, exclude_tables=True))
 
     def render_list(list_element: _Element, depth: int) -> list[str]:
         list_lines: list[str] = []