Feature/centralized test reporting issue 77 - #127
Merged
andytorrestb merged 8 commits intoJul 26, 2026
Merged
Conversation
Declare pytest-html and PyYAML as a PEP 735 dependency group rather than
[project.optional-dependencies]: PyRPOD has no [project] table because it
is not installed as a distribution, and a dependency group adds these
without introducing packaging metadata or changing install behavior.
python -m pip install --group test # pip >= 25.1
python -m pip install "pytest-html>=4.2,<5" "PyYAML>=6"
Also registers the `tooling` marker for the test-infrastructure tests
added later in this series, and ignores reports/, which holds
run-specific HTML reports and failed-test logs. The generated
tests/README.md and tests/test_manifest.yaml stay tracked.
Refs plume-kit#77
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Twelve collected tests exercise no observable behavior: the body is a bare
return, or the intended analysis is entirely commented out. They reported
as passed, which made an empty stub indistinguishable from real coverage
in any summary.
Mark each with pytest.mark.skip and a reason naming the manifest, so the
HTML report shows them as skipped. unittest.TestCase assertion methods and
numpy.testing helpers were checked before classifying, so tests that
assert without a plain `assert` are not caught up in this.
Purely additive: an import and a decorator per file. No test body,
assertion, input, expected value, or numerical baseline is touched.
mdao_unit_test_01/02, mdao_integration_test_01,
mdao_verification_test_04/05/06, mission_unit_test_01,
mission_integration_test_02, mission_verification_test_01,
plume_unit_test_01, plume_integration_test_01,
plume_verification_test_01
Refs plume-kit#77
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Single source of truth for inventory-level metadata about every test asset
under tests/ -- 93 entries covering collected pytest tests, manual
verification and figure-reproduction scripts, ignored/blocked files and
the archived tests/old/ suite.
Schema, one entry per file:
path, description, subsystem, category, execution_mode,
development_status, collection_status, reference, manual_command,
collection_ignore_reason
Development status (implemented / placeholder / needs_review / blocked /
archived / deprecated) is deliberately disjoint from execution outcome:
passed and failed are not valid values here, because a green run says
nothing about whether a test is finished. Ten tests that run real pipeline
code but assert nothing are recorded as needs_review rather than skipped,
so their execution coverage is kept while the gap stays visible.
Descriptions are seeded from the previous tests/README.md dashboard,
module header comments and docstrings. Ignore reasons come from the
existing collect_ignore configuration in tests/conftest.py.
Refs plume-kit#77
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One canonical command for the whole reporting workflow:
python scripts/generate_test_dashboard.py
It validates the manifest schema, collects the current pytest tests,
cross-checks the two, regenerates tests/README.md, then runs pytest with
pytest-html into reports/pyrpod-pytest-report.html. It exits with pytest's
own exit code and leaves the report in place when tests fail, so CI can
publish the artifact and still fail the job.
Validation fails loudly rather than silently omitting anything. A test
file pytest collects with no manifest entry, an entry claiming collection
that pytest no longer collects, and a manual script that has started
defining pytest tests are each reported by path with an explanation, and
tests/README.md is left untouched.
Cross-checking uses pytest's own collection output rather than parsing
test sources, so it cannot drift from what actually runs.
Two auxiliary modes keep the edit loop fast: --inventory-only regenerates
the README without running the suite, and --check verifies the committed
README is current without writing.
Refs plume-kit#77
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add pytest-html hooks that widen the results table with the inventory metadata a reader needs to interpret a result: description, subsystem, category, execution mode, development status, reference and source path. Standard pytest-html result filtering, tracebacks and captured stdout/stderr/log are retained. For failed tests only, PyRPOD run logs are associated by modification time inside that test's own call phase -- so a log can never be attributed to a test that did not write it -- then copied to reports/logs/ for the CI artifact and embedded in the report. The application logging system itself is unchanged. All of this degrades to a plain pytest-html report if PyYAML or the manifest is unavailable, and is inert without pytest-html. A bare `pytest` run must never depend on the reporting extras, which is also why placeholder skips live as markers in the test files rather than being driven from the manifest at collection time. Also registers `tooling` as a subsystem group so test-infrastructure tests get the same automatic marker treatment as the science subsystems. Refs plume-kit#77 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replace the hand-maintained status dashboard with a file generated from the manifest and live pytest collection, and add the tests that keep it honest. The two land together because they are mutually dependent: the generated README counts the tooling tests, and one tooling test asserts the committed README matches what the generator produces. The inventory summarizes counts, groups by subsystem with unit / integration / verification subsections, and gives manual scripts, placeholders, ignored/blocked files and the archived suite their own sections. Ignored entries carry the reason they are excluded; manual scripts carry their exact run command. No pass/fail result appears anywhere -- the legend spells out why execution outcome, development status and collection status are three independent axes. 27 tests cover schema validation, the controlled vocabularies (asserting that runtime outcomes such as `passed` are rejected as metadata), missing/stale entry detection, and deterministic rendering. Two use pytest's own collected items and their markers rather than parsing source: one proves every manifest placeholder is actually skipped and no other test is, the other proves every collected file is documented. Refs plume-kit#77 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Run the dashboard generator in place of bare `pytest` on every pull request and every push to master, and publish the result as the `pyrpod-test-report` artifact (14-day retention). The upload step is guarded with `if: always()` so a failing run still produces a report -- which is when the report is most useful. Because the generator returns pytest's exit code, the job still fails after the artifact is uploaded; no unconditional success handling hides a test failure. pip is upgraded before `pip install --group test`, since PEP 735 dependency groups need pip >= 25.1. GitHub Pages publishing is deliberately out of scope. Refs plume-kit#77 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add a Test Reporting section to the root README covering the dependency install, the dashboard command, where the HTML report is written, how the manifest and generated README relate, and how to reach the CI artifact. It leads with the distinction that motivates the whole design: a passing test is not evidence that a test is finished. Extend the CONTRIBUTING testing section with what a contributor now owes when adding a test -- a manifest entry and a regenerated tests/README.md -- and states that development status, never a pass/fail result, is what the manifest records. Kept to the two existing pages; no new overlapping documentation. Refs plume-kit#77 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.