Feature: Create an automated way to update reference traces#583
Open
Feature: Create an automated way to update reference traces#583
Conversation
Add a pytest CLI flag (--update-references) and environment variable (UPDATE_REFERENCE_TRACES=1) that cause regression tests to overwrite checked-in reference CSV directories with freshly generated outputs instead of comparing against them. This eliminates the manual workflow of finding generated outputs under /tmp or pytest temp directories and copying them into the tests/traces/ tree. Changes: - conftest.py: Add pytest_addoption hook, update_references fixture, and update_reference_csvs helper - test_perf_report_regression.py: Use update_references fixture - test_compare_perf_reports.py: Use update_references fixture (both basic and E2E tests) - test_jax_perf_report.py: Use update_references fixture - test_detect_recompute.py: Use update_references fixture - test_collective_analysis.py: Use update_references fixture Co-authored-by: Kyle Hoffmeyer <unagiboi@users.noreply.github.com>
ajassani
approved these changes
Apr 9, 2026
Collaborator
ajassani
left a comment
There was a problem hiding this comment.
We should add to the code comment/docstring as well as contributing.md that
when you use this flag, explicitly note in your PR description which references were updated and why. Do not use this flag to paper over unexpected differences.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds an automated way to update the checked-in reference CSV directories directly from the test runner.
What changed
New infrastructure in
tests/conftest.py--update-referencespytest CLI flag — pass to anypytestinvocation to enable update modeupdate_referencesfixture — injects a boolean into tests that checks both the CLI flag and the env varupdate_reference_csvs()helper — atomically replaces a reference CSV directory with freshly generated output (removes stale sheets)Updated test modules
All regression test modules that compare generated output against checked-in reference CSV directories now accept the
update_referencesfixture:test_perf_report_regression.py*_perf_report_csvs/directories next to.json.gztracestest_compare_perf_reports.pyreference/subdirectories for compare tests (basic + E2E)test_jax_perf_report.py*_perf_report_csvs/directories for JAX tracestest_detect_recompute.pyresnet_act_checkpoint_recompute_perf_report_csvs/test_collective_analysis.pynccl_analysis_report_csvs/Usage
When update mode is active, each test:
Closes #561