diff --git a/AGENTS.md b/AGENTS.md index dcdcf1f..9320524 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -112,7 +112,7 @@ The CLI commands include source acquisition, quality, assembly, analysis, and Fo | `loop_apidoc/gitbook_llms.py` | deterministic GitBook `llms.txt` filtering/cache with safe path preservation, URL sidecars, and coverage | | `loop_apidoc/markdown_drafts/` | separate non-authoritative, line-cited Markdown endpoint/table/example drafts; never alters `source_facts` validation | | `loop_apidoc/extraction_scaffold/` | pure projection of Markdown drafts into review-only extraction-shaped inventory/endpoint JSON; `write.py` is this feature's sole atomic output exit, and agents must copy/review output before it is used as real extraction | -| `loop_apidoc/html_snapshot.py` | `normalize-html-snapshot`: `html_to_markdown` (pure: readable main-document text, no invented content) + `normalize_html_snapshot` (writes the Markdown and a `.source.json` sidecar binding it to the raw file's URL + sha256) | +| `loop_apidoc/html_snapshot.py` | `normalize-html-snapshot`: `html_to_markdown` (pure: readable main-document text, no invented content; `colspan`/`rowspan` expand into a rectangular grid — the spanning cell's text stays in its own column and the covered columns are left blank so a group-title row stays distinguishable from a parameter row, while `rowspan` carries the text down its own column; out-of-range/non-numeric spans count as 1, overlapping spans discard the whole table, a multi-row `thead` merges into one header row, and a nested table renders as its own table instead of appending its rows to the enclosing one — a misaligned parameter table becomes a source fact nobody wrote) + `normalize_html_snapshot` (writes the Markdown and a `.source.json` sidecar binding it to the raw file's URL + sha256) | | `loop_apidoc/rendered_url.py` | first-class offline browser-rendered URL import: validates original/canonical URL, timezone-aware capture metadata, capture method, safe immutable destinations, and SHA-256; writes the unchanged HTML/Markdown source, versioned provenance sidecar, and `fetched_rendered` coverage. Its read-side verifier binds coverage → sidecar → manifest local source and fails closed on any mismatch before an origin probe. | | `loop_apidoc/source_risk/` | deterministic pre-agent gate behind `inspect-source-risk`: `models.py` (versioned `SourceRiskReport`, findings and coverage), `inspect.py` (bounded read of manifest-bound UTF-8 Markdown/HTML/OpenAPI JSON/YAML; fixed rules, a 1,000-entry report cap with fail-closed `SR-FINDINGS-TRUNCATED`, and stable `source_binding_digest`; PDF/Word, invalid UTF-8, over-`max_bytes`, and other unscannable pending sources are blockers), `loader.py` (fail-loud schema/ruleset/verdict/manifest/source-binding verification plus deterministic reinspection of current bytes), `report.py` (`source-risk-report.{json,zh-TW.md}`; findings never echo matched payloads). Exit 0/1/2 means pass/reject/input error; source bytes are never rewritten. | | `loop_apidoc/source_quality/` | pre-extraction source quality gate behind `assess-sources --source-risk`: `models.py` (`QualityObservation`/`QualityFinding`, `FindingSeverity`, verdict `pass`/`reject` + `SourceDiffReport`; report embeds a verified `SourceRiskReport`; blocker observations may carry explicit HTTP(S) `required_source_refs`), `loader.py` (read side: manifest, agent-written observations JSON, and a completed assessment dir — `SourceQualityInputError`), `assess.py` (`assess_source_quality`, pure: manifest usability + observations + source-risk audit → findings; any blocker ⇒ `reject`; rejected reports aggregate a bounded ordered/de-duplicated `required_source_refs` list without fetching it), `diff.py` (`build_source_diff`, pure manifest-vs-manifest added/removed/changed), `report.py` (`write_reports` → `source-quality-report.{json,zh-TW.md}` + `source-diff.{json,md}`). `assemble` requires `--source-quality`, re-loads the reports, rejects missing/stale/mismatched embedded audits and internally inconsistent reports (strict `extra="forbid"` models; `verdict` must be `reject` exactly when a blocker finding exists, and `required_source_refs` must equal the ordered de-duplicated union from those blockers), and copies a passing pair into the run-dir's `source-quality/`. | diff --git a/docs/HTML_SNAPSHOT_NORMALIZATION_SPEC.md b/docs/HTML_SNAPSHOT_NORMALIZATION_SPEC.md index f05ddf0..ae9273c 100644 --- a/docs/HTML_SNAPSHOT_NORMALIZATION_SPEC.md +++ b/docs/HTML_SNAPSHOT_NORMALIZATION_SPEC.md @@ -45,6 +45,15 @@ policy changes in this work. their descendants. Do not add heuristic content scoring or site-specific rules. 3. Preserve the existing output for headings, paragraphs, table cells (including escaped pipes), and `pre`/`code` block line breaks. +3a. Render each table on its own grid. A table owns only its own rows: a nested + table becomes a separate table block after its parent, and its text is not pulled + into the enclosing cell. Expand `colspan`/`rowspan` into a rectangular grid — the + spanning cell's text stays in its own column and the columns it covers are left + blank, while `rowspan` carries the text down its own column. Do not repeat a + spanning cell across the columns it covers: a row whose remaining cells are blank + is how downstream fact extraction tells a group-title row from a parameter row. + Out-of-range or non-numeric spans count as 1, overlapping spans discard the whole + table, and a multi-row `thead` merges into the single header row GFM allows. 4. Render nested unordered and ordered lists as Markdown list items with deterministic indentation. Preserve each item's readable text once; group/empty wrapper nodes must not produce duplicate lines. Identical sibling items remain distinct: source diff --git a/loop_apidoc/html_snapshot.py b/loop_apidoc/html_snapshot.py index c3909bb..31cd1bc 100644 --- a/loop_apidoc/html_snapshot.py +++ b/loop_apidoc/html_snapshot.py @@ -5,10 +5,170 @@ import json from datetime import datetime, timezone from hashlib import sha256 +from collections.abc import Callable from pathlib import Path from loop_apidoc.url_catalog import _Element, _TreeParser, _walk +#: How far a cell may credibly span. Real parameter tables stay well inside these; +#: anything larger is broken markup, and expanding it verbatim invents a table shape +#: nobody wrote. +_MAX_COLSPAN = 16 +_MAX_ROWSPAN = 64 + + +def _span(cell: _Element, attribute: str, limit: int) -> int: + """A span the document actually states, or 1. + + Out-of-range and non-numeric values fall back to 1 rather than being clamped to + the limit: `colspan="9999"` is a broken document, and honouring it as "the widest + we allow" would invent a shape nobody wrote. + """ + try: + value = int(cell.attrs.get(attribute, "1")) + except ValueError: + return 1 + return value if 1 <= value <= limit else 1 + + +def _own_rows(table: _Element) -> tuple[list[_Element], list[_Element]]: + """This table's own rows as (header, body), nested tables left to themselves. + + A nested table's rows belong to it; collecting every descendant `tr` appends them + to the enclosing table, where they line up against the wrong columns — and a + misaligned parameter table becomes a source fact the source never stated. + + Rows are also collected from inside a `tr`, because the parser implements no + implied end tags: a source that omits `` nests every following row inside the + first one, and refusing to descend would drop the whole table body. + """ + head: list[_Element] = [] + body: list[_Element] = [] + foot: list[_Element] = [] + + def visit(node: _Element, *, section: list[_Element]) -> None: + for child in node.children: + if not isinstance(child, _Element) or child.tag == "table": + continue + if child.tag == "tr": + section.append(child) + visit(child, section=section) + continue + if child.tag == "thead": + visit(child, section=head) + elif child.tag == "tfoot": + visit(child, section=foot) + else: + visit(child, section=section) + + visit(table, section=body) + return head, body + foot + + +def _nested_tables(table: _Element) -> list[_Element]: + """The outermost tables inside this one; each renders itself recursively.""" + found: list[_Element] = [] + + def visit(node: _Element) -> None: + for child in node.children: + if not isinstance(child, _Element): + continue + if child.tag == "table": + found.append(child) + else: + visit(child) + + visit(table) + return found + + +def _table_grid( + rows: list[_Element], + cell_text: Callable[[_Element], str], +) -> list[list[str]]: + """Expand `colspan`/`rowspan` into a rectangular grid, or `[]` if it cannot. + + A spanning cell's text stays at its own position and the columns it covers are + left blank. Repeating it across those columns would be the alignment fix that + manufactures facts: downstream, a row whose remaining cells are blank is how a + group-title row ("Header", "支付類") is told apart from a parameter row, and a + repeated title fills those cells with a value the source never stated. + + `rowspan` is the exception and repeats down its own column: there the carried + text is a real value for each of the rows it covers, and dropping it would strip + the group a nested field belongs to. + + Overlapping spans return `[]` so the caller renders nothing — silently letting a + later cell overwrite a carried one produces a table that looks fine and is wrong, + the same bias the error-code reader takes when one row is malformed. + """ + cells: dict[tuple[int, int], str] = {} + owners: dict[tuple[int, int], tuple[int, int]] = {} + for index, row in enumerate(rows): + column = 0 + for cell in row.children: + if not isinstance(cell, _Element) or cell.tag not in {"th", "td"}: + continue + while (index, column) in cells: + column += 1 + text = cell_text(cell).replace("|", r"\|") + columns = _span(cell, "colspan", _MAX_COLSPAN) + down = min(_span(cell, "rowspan", _MAX_ROWSPAN), len(rows) - index) + for offset in range(down): + for shift in range(columns): + position = (index + offset, column + shift) + if owners.get(position, (index, column)) != (index, column): + return [] + owners[position] = (index, column) + cells[position] = text if shift == 0 else "" + column += columns + if not cells: + return [] + width = max(column for _, column in cells) + 1 + height = max(index for index, _ in cells) + 1 + return [ + [cells.get((index, column), "") for column in range(width)] + for index in range(height) + ] + + +def _merge_header(rows: list[list[str]]) -> list[str]: + """Fold a multi-row `thead` into the single header row GFM allows. + + Demoting the extra rows to the body instead would hand the fact scanner a + parameter row built out of column titles. + """ + merged: list[str] = [] + for column in range(len(rows[0])): + parts: list[str] = [] + for row in rows: + value = row[column] + if value and value not in parts: + parts.append(value) + merged.append(" ".join(parts)) + return merged + + +def _render_table(table: _Element, cell_text: Callable[[_Element], str]) -> str: + head, body = _own_rows(table) + grid = _table_grid(head + body, cell_text) + blocks: list[str] = [] + if grid: + split = max(len(head), 1) + header = _merge_header(grid[:split]) if split > 1 else grid[0] + lines = [ + "| " + " | ".join(header) + " |", + "| " + " | ".join(["---"] * len(header)) + " |", + ] + lines += ["| " + " | ".join(row) + " |" for row in grid[split:]] + blocks.append("\n".join(lines)) + blocks += [ + rendered + for rendered in (_render_table(nested, cell_text) for nested in _nested_tables(table)) + if rendered + ] + return "\n\n".join(blocks) + def html_to_markdown(html: str) -> str: """Extract readable main-document text without inventing content.""" @@ -47,7 +207,12 @@ def visit(node: _Element | str) -> None: visit(item) return " ".join("".join(parts).split()) - def inline_text(item: _Element, *, exclude_lists: bool = False) -> str: + def inline_text( + item: _Element, + *, + exclude_lists: bool = False, + exclude_tables: bool = False, + ) -> str: """Render readable inline content without resolving or inventing links.""" parts: list[str] = [] @@ -57,6 +222,11 @@ def visit(node: _Element | str) -> None: return if node.tag in ignored or (exclude_lists and node.tag in {"ul", "ol"}): return + # A nested table is rendered as its own table, so pulling its text into + # the enclosing cell would state the same rows twice — once as a run-on + # sentence that no longer says which value belongs to which column. + if exclude_tables and node.tag == "table": + return if node.tag == "a": label = plain_text(node) href = node.attrs.get("href", "") @@ -80,26 +250,7 @@ def raw_text(item: _Element) -> str: return "".join(parts) def render_table(table: _Element) -> str: - rows: list[list[str]] = [] - for row in (e for e in _walk(table) if e.tag == "tr"): - cells = [ - inline_text(cell).replace("|", r"\|") - for cell in row.children - if isinstance(cell, _Element) and cell.tag in {"th", "td"} - ] - if cells: - rows.append(cells) - if not rows: - return "" - width = max(len(row) for row in rows) - rows = [row + [""] * (width - len(row)) for row in rows] - header, *body = rows - out = [ - "| " + " | ".join(header) + " |", - "| " + " | ".join(["---"] * width) + " |", - ] - out += ["| " + " | ".join(row) + " |" for row in body] - return "\n".join(out) + return _render_table(table, lambda cell: inline_text(cell, exclude_tables=True)) def render_list(list_element: _Element, depth: int) -> list[str]: list_lines: list[str] = [] diff --git a/tests/test_html_snapshot_tables.py b/tests/test_html_snapshot_tables.py new file mode 100644 index 0000000..483b02d --- /dev/null +++ b/tests/test_html_snapshot_tables.py @@ -0,0 +1,260 @@ +"""HTML 表格轉 Markdown 的結構保真度(#81)。 + +參數表是來源事實的主要來源,而來源事實會被 fail-closed 閘門當成「必須被擷取」的 +證據。表格錯位一格,擷取就會被要求交出來源根本沒寫的欄位——假事實比漏掉事實貴, +因為它擋掉的是正確的擷取。 + +seam 是 `html_to_markdown` 這個公開純函式。 +""" +from __future__ import annotations + +from loop_apidoc.html_snapshot import html_to_markdown + + +def _rows(md: str) -> list[list[str]]: + return [ + [cell.strip() for cell in line.strip().strip("|").split("|")] + for line in md.splitlines() + if line.startswith("|") and "---" not in line + ] + + +def test_colspan_keeps_the_following_columns_aligned(): + """跨欄的表頭不得把後面每一欄都往左推一格。""" + html = ( + "
" + "" + "" + "
RequestNote
namestring必填
" + ) + + rows = _rows(html_to_markdown(html)) + + # 文字只留在它自己的位置,跨到的欄位補空:重複它會毀掉「其餘欄位全空」這個 + # 下游用來認出分組標題列的訊號(見 source_facts/markdown.py)。 + assert rows[0] == ["Request", "", "Note"] + assert rows[1] == ["name", "string", "必填"] + + +def test_rowspan_carries_the_cell_down_its_own_column(): + """跨列的第一欄不得讓下面每一列都少一格。""" + html = ( + "
" + "" + "" + "" + "
GroupNameType
cardholdernameString
emailString
" + ) + + rows = _rows(html_to_markdown(html)) + + assert rows[1] == ["cardholder", "name", "String"] + assert rows[2] == ["cardholder", "email", "String"] + + +def test_a_nested_table_does_not_become_rows_of_the_outer_table(): + """內層表格的列併進外層,就是把兩張表的欄位對錯位。""" + html = ( + "
" + "" + "" + "
NameUsage
cardholder" + "" + "
SubType
phoneString
" + "
" + ) + + md = html_to_markdown(html) + outer, _, nested = md.partition("\n\n| Sub | Type |") + + assert nested, "內層表格必須是自己的區塊,而不是外層表格的續列" + assert "phone" not in outer + assert "| Name | Usage |" in outer + + +def test_a_nested_table_is_still_rendered_as_its_own_table(): + """不併進外層,不等於丟掉——內層表格照樣是來源寫過的參數表。""" + html = ( + "
" + "" + "" + "
NameUsage
cardholder" + "" + "
SubType
phoneString
" + "
" + ) + + rows = _rows(html_to_markdown(html)) + + assert ["Sub", "Type"] in rows + assert ["phone", "String"] in rows + + +def test_the_outer_cell_does_not_swallow_the_nested_table_text(): + """內層表格的內容不該同時以跑馬燈文字塞回外層儲存格。""" + html = ( + "
" + "" + "" + "
NameUsage
cardholderHolder info" + "
phoneString
" + "
" + ) + + rows = _rows(html_to_markdown(html)) + + assert ["cardholder", "Holder info"] in rows + + +def test_a_thead_row_is_the_header_even_when_it_is_not_first_in_source_order(): + html = ( + "
" + "" + "" + "
actionY
參數必要
" + ) + + rows = _rows(html_to_markdown(html)) + + assert rows[0] == ["參數", "必要"] + assert ["action", "Y"] in rows[1:] + + +def test_a_body_only_table_still_uses_its_first_row_as_the_header(): + """GFM 沒有無表頭的表格,既有行為維持不變。""" + html = ( + "
" + "" + "" + "
actionY
tsN
" + ) + + rows = _rows(html_to_markdown(html)) + + assert rows[0] == ["action", "Y"] + + +def test_an_absurd_span_is_ignored_rather_than_expanded(): + """`colspan="9999"` 展開會生出一張沒人寫過的表;寧可當成 1。""" + html = ( + "
" + "" + "" + "
NameType
nameString
" + ) + + rows = _rows(html_to_markdown(html)) + + assert rows[1] == ["name", "String"] + + +def test_a_non_numeric_span_is_treated_as_one(): + html = ( + "
" + "" + "" + "
NameType
nameString
" + ) + + rows = _rows(html_to_markdown(html)) + + assert rows[1] == ["name", "String"] + + +def test_a_colspan_group_title_row_stays_recognisable_as_a_group_title(): + """跨欄的分組標題列不得變成一個叫「Header」的參數。 + + `source_facts/markdown.py` 認分組標題列的判準是「其餘欄位全空」。把跨欄儲存格 + 的文字重複填進每一欄會毀掉那個訊號,於是來源根本沒寫的欄位變成來源事實,而 + 假事實在 fail-closed 閘門下會擋掉正確的擷取——正是這張票要防的傷害。 + """ + from loop_apidoc.source_facts.markdown import scan_markdown + + html = ( + "

POST /pay

" + "" + "" + "" + "
參數型別必要
Header
api_keyStringY
" + ) + + facts = scan_markdown("doc.md", html_to_markdown(html)) + + assert facts.endpoints[0].parameter_names == ["api_key"] + + +def test_a_spanning_section_row_does_not_void_an_error_code_table(): + """錯誤碼表的分組列同理:訊號一毀,整張表被作廢,記載下界靜靜歸零。""" + from loop_apidoc.source_facts.markdown import scan_markdown + + html = ( + "

錯誤碼

" + "" + "" + "" + "
錯誤碼說明
支付類
1001餘額不足
" + ) + + facts = scan_markdown("doc.md", html_to_markdown(html)) + + assert [fact.code for fact in facts.error_codes] == ["1001"] + + +def test_rows_survive_a_missing_closing_tr_tag(): + """`` 未閉合時解析器會把下一列變成上一列的子節點,那些列不得消失。""" + html = ( + "
" + "" + "" + "
ab
cd
" + ) + + rows = _rows(html_to_markdown(html)) + + assert rows == [["a", "b"], ["c", "d"]] + + +def test_overlapping_spans_discard_the_whole_table(): + """對不齊就整張放棄,與錯誤碼表「一列壞掉作廢整張」的既有偏誤一致。 + + 靜靜讓後寫的儲存格蓋掉前一列帶下來的,會產出一張看起來正常、內容卻錯位的表。 + """ + html = ( + "
" + "" + "" + "
AB
C
" + ) + + assert _rows(html_to_markdown(html)) == [] + + +def test_a_tfoot_written_before_tbody_still_renders_after_it(): + """HTML 允許 tfoot 寫在 tbody 之前,但它是表尾。""" + html = ( + "
" + "" + "" + "" + "
NameType
f1f2
b1b2
" + ) + + rows = _rows(html_to_markdown(html)) + + assert rows == [["Name", "Type"], ["b1", "b2"], ["f1", "f2"]] + + +def test_a_two_row_thead_merges_into_one_header(): + """GFM 只有一列表頭。第二列降級成資料列會變成一個叫「Name」的參數。""" + html = ( + "
" + "" + "" + "" + "
RequestRequest
NameType
api_keyString
" + ) + + rows = _rows(html_to_markdown(html)) + + assert rows[0] == ["Request Name", "Request Type"] + assert rows[1] == ["api_key", "String"]