diff --git a/architecture.md b/architecture.md index c270295..c6a43f7 100644 --- a/architecture.md +++ b/architecture.md @@ -77,7 +77,11 @@ Most-touched top-level directories: - `harness/compat_cases.py`: openpyxl-style compatibility snippet lane - `harness/artifact_context.py`: chart and macro package evidence lanes - `harness/mutation.py`: template-mutation lane (subprocess wall time, RSS, - package-preservation scoring) + content-model preservation scoring; see DEC-027) + - `harness/content_model_xlsx.py`: xlsx content model used to score mutation + preservation + - `harness/adapters/json_model_adapter.py`: base for adapters driven through + an external JSON helper (SheetJS, ExcelJS, LibreOffice) - `harness/calc.py`: formula-recalculation tier engines (wolfxl, LibreOffice, aspose-cells-foss, zavora oracle) against the cache-free fixture - `modifiable.py`: engine registry for mutation-capable adapters @@ -95,14 +99,18 @@ Most-touched top-level directories: - `calc/`: cache-free formula fixture + LibreOffice oracle expected values - `tools/external-oracles/` - - Optional subprocess helpers for non-Python oracle tools. + - Optional subprocess helpers for non-Python oracle tools and for the + JSON-model adapters (`sheetjs`, `exceljs`, `libreoffice`). - `excelize/`: Go helper that generates and inspects `.xlsx` fixtures with Excelize. Run from that directory with `go run .`. - `libreoffice/`: Python helper that runs LibreOffice headless open/save and - PDF render validation. + PDF render validation, plus the UNO macro behind the `libreoffice` adapter. - `exceljs/`: Node helper that generates workbooks with ExcelJS tables, formulas, data validations, rich strings, comments, hyperlinks, images, - merges, freeze panes, and sheet protection. + merges, freeze panes, and sheet protection, and serves the `exceljs` + adapter's model operations. + - `sheetjs/`: Node helper behind the `sheetjs` adapter (SheetJS Community + Edition). - `apache-poi/`: Java helper that generates workbooks with Apache POI tables, formulas, data validations, rich strings, comments, hyperlinks, images, merges, freeze panes, and sheet protection. @@ -143,7 +151,8 @@ Common starting points by intent: - Add a new external oracle: - `src/excelbench/harness/external_oracles.py` - `docs/trackers/external-oracle-expansion.md` - - Keep helpers optional and subprocess-isolated until promoted into normal benchmark flows. + - Keep helpers optional and subprocess-isolated. A helper that implements the + JSON-model protocol is registered as a normal adapter (DEC-027). - Add a new scored feature: - Generator: `src/excelbench/generator/features/` @@ -200,9 +209,12 @@ JSON fixture manifest -> optional WolfXL validation script checks read + in-place modify-save preservation ``` -External oracle helpers are intentionally not part of `get_all_adapters()` yet. -Missing Go/Java/.NET/LibreOffice commands should skip cleanly, not fail the core -suite. The first local pack is generated by +External helpers that implement the JSON-model protocol (`sheetjs`, `exceljs`, +`libreoffice`) are registered in `get_all_adapters()` and scored like any other +adapter; `apache-poi`, `excelize` and `zavora-xlsx` are registered the same way +and run in the cross-language lane (DEC-027). Registration is availability-gated: +a missing Node/Go/Java/.NET/LibreOffice runtime or helper dependency removes the +adapter instead of failing the core suite. The first local oracle pack is generated by `scripts/generate_external_oracle_fixtures.py` into `results_dev_external/` and validated against WolfXL with `scripts/validate_external_oracle_fixtures_with_wolfxl.py`. diff --git a/decisions.md b/decisions.md index f8ffe66..a1a0c46 100644 --- a/decisions.md +++ b/decisions.md @@ -40,6 +40,57 @@ Skip logging for routine bug fixes, refactors, or incremental test additions. ## Decisions +### DEC-027 — External-helper adapters are scored; mutation preservation compares parsed content (2026-10-02) + +**Context**: DEC-021 kept external helpers out of `get_all_adapters()` until a +later decision promoted them. The 2026-10-02 snapshot adds SheetJS, ExcelJS +and LibreOffice Calc, which have no Python API: each is driven by a helper in +`tools/external-oracles/` through the JSON-model protocol in +`harness/adapters/json_model_adapter.py`. Separately, the DEC-025 mutation +score counted retained package parts and byte-identical custom XML, so a byte +copy scored 100 by construction and an engine that rewrote the package +equivalently (new part names, renumbered relationship ids, other XML +serialization) lost points without losing any content. + +**Decision**: + +- `sheetjs`, `exceljs` and `libreoffice` are registered in + `get_all_adapters()` and scored like any other adapter in the fidelity and + mutation lanes, so the comparison covers non-Python libraries. Registration + is availability-gated: a missing Node or `soffice` runtime removes the + adapter instead of failing the run. Helpers report only what the library's + own API exposes, and the Python side never fills in values the helper did + not report. The same availability-gated registration already applied to + `apache-poi`, `excelize` and `zavora-xlsx`, which stay in the + cross-language lane (`CROSS_LANGUAGE_ADAPTER_NAMES`). +- Mutation preservation compares parsed content: the template's xlsx content + model (`harness/content_model_xlsx.py`) with the declared edits applied is + diffed against the output's model, plus a `custom_xml` feature (canonical + XML, datastore item id, schema references). Part names, relationship ids, + XML serialization and encoding, and document metadata are not content, so + an equivalent rewrite scores the same as an in-place edit. Cached results of + formula cells are recalculation state and are not compared; formula text is. + The result is features preserved (`kept/present`), edits applied, and an + unscored integrity report (dangling relationships, parts without a content + type). This supersedes the DEC-025 package-preservation score. + +**Alternatives considered**: + +1. **Keep JSON-model helpers in a separate context lane** — rejected. The + fidelity matrix would stay Python-only while the libraries users compare + against WolfXL and openpyxl include JavaScript and LibreOffice. +2. **Keep the part-retention score and whitelist known renames** — rejected. + Every new engine would need its own whitelist, and a byte copy would still + score perfectly without proving the edits landed. + +**Consequences**: + +- Fidelity and mutation results depend on which helper runtimes are + installed; snapshot READMEs record runtime versions and lane membership. +- Mutation scores before 2026-10-02 are not comparable with later scores. +- Package-level differences remain visible as unscored diagnostics (lost and + added parts, element changes) so layout regressions are still reviewable. + ### DEC-023 — Tier-4 fidelity features via openpyxl-structural fixtures (2026-09-08) **Context**: The scored matrix stopped at 19 features. Sheet protection, page @@ -137,6 +188,8 @@ cells, save, keep everything else comparable. - Preservation is a structural proxy, not a semantic one. The verifier reopens outputs with openpyxl to catch the failure class that matters (corrupt relationships, as zavora 0.1.2 demonstrates). +- The package-preservation score is superseded by content-model scoring in + DEC-027. ### DEC-026 — Calc tier: cache-free fixture, LibreOffice oracle, strict status semantics (2026-09-08) @@ -257,7 +310,8 @@ ExcelBench install or CI job. - The first oracle sprint can focus on helper binaries/scripts and fixture truth-passing without changing benchmark scoring. - Promotion into public results requires a later explicit decision once the - generated workbooks are deterministic and manually audited. + generated workbooks are deterministic and manually audited. DEC-027 records + that decision for the JSON-model adapters. **Commit(s)**: this commit. diff --git a/src/excelbench/harness/mutation.py b/src/excelbench/harness/mutation.py index a4a7f7e..6ee5fa1 100644 --- a/src/excelbench/harness/mutation.py +++ b/src/excelbench/harness/mutation.py @@ -183,14 +183,13 @@ def _missing_content_types(parts: dict[str, bytes]) -> list[str]: def _canonical_xml(data: bytes) -> str: - """Return C14N 2.0 text that ignores encoding, prefixes, attribute order and padding.""" + """Return C14N 2.0 text that ignores encoding, prefixes, attribute order and padding. + + The raw bytes go to the parser so the BOM and XML declaration select the decoding. + """ try: - return ET.canonicalize( - xml_data=data.decode("utf-8-sig"), - strip_text=True, - rewrite_prefixes=True, - ) - except (ET.ParseError, UnicodeDecodeError): + return ET.canonicalize(xml_data=data, strip_text=True, rewrite_prefixes=True) + except ET.ParseError: return f"unparsable:{hashlib.sha256(data).hexdigest()}" diff --git a/src/excelbench/results/html_dashboard.py b/src/excelbench/results/html_dashboard.py index 499b46e..825c52c 100644 --- a/src/excelbench/results/html_dashboard.py +++ b/src/excelbench/results/html_dashboard.py @@ -73,6 +73,20 @@ ] +def _scored_results(results: list[dict[str, Any]]) -> list[dict[str, Any]]: + """Drop rows of features no adapter scored (read and write null for every adapter). + + Such a feature is not in any denominator; counting it would charge every + library for a feature the run could not score. + """ + scored = { + e["feature"] + for e in results + if any(e.get("scores", {}).get(op) is not None for op in ("read", "write")) + } + return [e for e in results if e["feature"] in scored] + + def _compute_radar_data( fidelity: dict[str, Any], perf: dict[str, Any] | None, @@ -88,7 +102,7 @@ def _compute_radar_data( - Feature Coverage: features_with_any_score / total_features * 100 - Capability Breadth: R-only/W-only=33, R+W=66, R+W+Patch=100 """ - results = fidelity.get("results", []) + results = _scored_results(fidelity.get("results", [])) libs_info = fidelity.get("libraries", {}) focus = focus_libs or _DEFAULT_RADAR_LIBS available = set(libs_info.keys()) @@ -1404,8 +1418,9 @@ def _section_overview(fidelity: dict[str, Any], perf: dict[str, Any] | None) -> meta = fidelity.get("metadata", {}) libs = fidelity.get("libraries", {}) results = fidelity.get("results", []) + scored = _scored_results(results) - all_features = sorted({e["feature"] for e in results}) + all_features = sorted({e["feature"] for e in scored}) total_libs = len(libs) total_feats = len(all_features) @@ -1424,11 +1439,11 @@ def _section_overview(fidelity: dict[str, Any], perf: dict[str, Any] | None) -> # Green count green = sum( - 1 for e in results + 1 for e in scored if max((s for s in [e["scores"].get("read"), e["scores"].get("write")] if s is not None), default=-1) == 3 ) - total_scored = len(results) + total_scored = len(scored) cards = [ (str(total_libs), "Libraries Tested"), @@ -1449,11 +1464,11 @@ def _section_overview(fidelity: dict[str, Any], perf: dict[str, Any] | None) -> # WolfXL hero stats wolf_green = sum( - 1 for e in results if e["library"] == "wolfxl" + 1 for e in scored if e["library"] == "wolfxl" and max((s for s in [e["scores"].get("read"), e["scores"].get("write")] if s is not None), default=-1) == 3 ) - wolf_scored = sum(1 for e in results if e["library"] == "wolfxl") + wolf_scored = sum(1 for e in scored if e["library"] == "wolfxl") wolf_hero = "" if wolf_scored > 0: # Compute throughput headline from perf data if available @@ -1476,7 +1491,7 @@ def _section_overview(fidelity: dict[str, Any], perf: dict[str, Any] | None) -> f'
\U0001F43A
' f'
' f'

WolfXL at a glance (maintained by SynthGL)

' - f'

{wolf_green}/{wolf_scored} features at full fidelity (score 3)' + f'

{wolf_green}/{wolf_scored} features scored 3 in read or write' f'{wolf_perf_note}. ' f'Hybrid Rust+Python engine with an openpyxl-compatible API. ' f' assert rewritten_score["diagnostics"]["lost_parts"] # renamed parts are only diagnostics +def test_utf16_custom_xml_scores_the_same_as_in_place_edit(tmp_path: Path) -> None: + """Re-encoding customXml as UTF-16 with a matching declaration is not a content change.""" + in_place = _write_parts(tmp_path / "in_place.xlsx", _correct_output()) + parts = _correct_output() + item = "customXml/item1.xml" + _replace(parts, item, b'encoding="UTF-8"', b'encoding="UTF-16"') + parts[item] = parts[item].decode("utf-8").encode("utf-16") + utf16 = _write_parts(tmp_path / "utf16.xlsx", parts) + + in_place_score = score_preservation(TEMPLATE, in_place, MUTATIONS) + utf16_score = score_preservation(TEMPLATE, utf16, MUTATIONS) + + assert utf16_score["features_changed"] == {} + assert utf16_score["preserved"] is in_place_score["preserved"] is True + assert utf16_score["feature_preservation"] == in_place_score["feature_preservation"] + + @pytest.mark.parametrize( ("drop", "feature"), [ diff --git a/tests/test_results_html_dashboard.py b/tests/test_results_html_dashboard.py index 0586179..96c0405 100644 --- a/tests/test_results_html_dashboard.py +++ b/tests/test_results_html_dashboard.py @@ -2,7 +2,11 @@ from pathlib import Path -from excelbench.results.html_dashboard import _compute_radar_data, render_html_dashboard +from excelbench.results.html_dashboard import ( + _compute_radar_data, + _section_overview, + render_html_dashboard, +) def test_render_html_dashboard_smoke(tmp_path: Path) -> None: @@ -111,6 +115,45 @@ def test_compute_radar_data_uses_p50_when_op_count_missing() -> None: assert by_lib["pandas"][2] == 25.0 +def test_feature_no_adapter_scored_is_excluded_from_denominators() -> None: + """A feature null for every adapter leaves the denominators; one scored by any stays.""" + fidelity = { + "libraries": { + "wolfxl": {"capabilities": ["read", "write", "modify"]}, + "openpyxl": {"capabilities": ["read", "write"]}, + }, + "results": [ + {"feature": "cell_values", "library": "wolfxl", "scores": {"read": 3, "write": 3}}, + {"feature": "cell_values", "library": "openpyxl", "scores": {"read": 3, "write": 3}}, + {"feature": "formulas", "library": "wolfxl", "scores": {"read": 1, "write": None}}, + { + "feature": "formulas", + "library": "openpyxl", + "scores": {"read": None, "write": None}, + }, + { + "feature": "pivot_tables", + "library": "wolfxl", + "scores": {"read": None, "write": None}, + }, + { + "feature": "pivot_tables", + "library": "openpyxl", + "scores": {"read": None, "write": None}, + }, + ], + } + + html = _section_overview(fidelity, None) + + assert "1/2 features scored 3 in read or write" in html + assert '

2
Features Scored
' in html + assert '
2/4
Score\u20033 Results
' in html + rows = _compute_radar_data(fidelity, None, focus_libs=["wolfxl", "openpyxl"]) + coverage = {r["library"]: r["values"][3] for r in rows} + assert coverage == {"wolfxl": 100.0, "openpyxl": 50.0} + + def test_render_html_dashboard_shows_delta_and_unsupported(tmp_path: Path) -> None: fidelity = tmp_path / "results.json" fidelity.write_text(