Skip to content

Feature: Create an automated way to update reference traces#583

Open
unagiboi wants to merge 4 commits intomainfrom
cursor/trace-update-automation-scope-1a9b
Open

Feature: Create an automated way to update reference traces#583
unagiboi wants to merge 4 commits intomainfrom
cursor/trace-update-automation-scope-1a9b

Conversation

@unagiboi
Copy link
Copy Markdown
Collaborator

@unagiboi unagiboi commented Apr 4, 2026

Summary

This PR adds an automated way to update the checked-in reference CSV directories directly from the test runner.

What changed

New infrastructure in tests/conftest.py

  • --update-references pytest CLI flag — pass to any pytest invocation to enable update mode
  • update_references fixture — injects a boolean into tests that checks both the CLI flag and the env var
  • update_reference_csvs() helper — atomically replaces a reference CSV directory with freshly generated output (removes stale sheets)

Updated test modules

All regression test modules that compare generated output against checked-in reference CSV directories now accept the update_references fixture:

Test file What it updates
test_perf_report_regression.py *_perf_report_csvs/ directories next to .json.gz traces
test_compare_perf_reports.py reference/ subdirectories for compare tests (basic + E2E)
test_jax_perf_report.py *_perf_report_csvs/ directories for JAX traces
test_detect_recompute.py resnet_act_checkpoint_recompute_perf_report_csvs/
test_collective_analysis.py nccl_analysis_report_csvs/

Usage

# Update all references at once
pytest tests/ --update-references

# Update only PyTorch perf report references
pytest tests/test_perf_report_regression.py --update-references

# Update references used by specific test
pytest tests/test_compare_perf_reports.py::test_compare_perf_reports --update-references

When update mode is active, each test:

  1. Generates the report as usual
  2. Copies the generated CSV directory over the checked-in reference directory

Closes #561

cursoragent and others added 4 commits April 3, 2026 19:31
Add a pytest CLI flag (--update-references) and environment variable
(UPDATE_REFERENCE_TRACES=1) that cause regression tests to overwrite
checked-in reference CSV directories with freshly generated outputs
instead of comparing against them.

This eliminates the manual workflow of finding generated outputs
under /tmp or pytest temp directories and copying them into the
tests/traces/ tree.

Changes:
- conftest.py: Add pytest_addoption hook, update_references fixture,
  and update_reference_csvs helper
- test_perf_report_regression.py: Use update_references fixture
- test_compare_perf_reports.py: Use update_references fixture (both
  basic and E2E tests)
- test_jax_perf_report.py: Use update_references fixture
- test_detect_recompute.py: Use update_references fixture
- test_collective_analysis.py: Use update_references fixture

Co-authored-by: Kyle Hoffmeyer <unagiboi@users.noreply.github.com>
@unagiboi unagiboi marked this pull request as ready for review April 6, 2026 16:54
@unagiboi unagiboi requested review from ajassani and gabeweisz April 6, 2026 16:54
@unagiboi unagiboi added the enhancement New feature or request label Apr 6, 2026
Copy link
Copy Markdown
Collaborator

@ajassani ajassani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add to the code comment/docstring as well as contributing.md that
when you use this flag, explicitly note in your PR description which references were updated and why. Do not use this flag to paper over unexpected differences.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Create an automated way to update reference traces

3 participants