Skip to content

Refresh training reports before post-upload scripts - #3214

Merged
bghira merged 1 commit into
mainfrom
fix/3207-refresh-upload-report
Sep 6, 2026
Merged

bghira merged 1 commit into
mainfrom
fix/3207-refresh-upload-report

Conversation

@bghira

@bghira bghira commented Sep 6, 2026

Copy link
Copy Markdown
Owner

The static report embeds a snapshot refreshed every 100 metric records. Post-upload scripts could therefore copy an HTML report from step 943 even after metrics and validation images for checkpoint 1000 had been written.

Refresh the enabled local report before submitting an external hook. This reuses the shared path used by trainer uploads, validation publishing, and checkpoint hooks, so the report includes the latest recorded metrics, validation media, and timestep samples. Serialize report renders to prevent periodic and background upload refreshes from sharing a temporary file or overwriting a newer snapshot with an older one. Runs without the local tracker do not create reports.

Validation:

  • The new hook regression failed before the fix with report step 943 instead of 1002, then passed with checkpoint-1000 validation media included. The hook's current-step placeholder remains 1002.
  • The concurrent-render regression failed before the fix with an older snapshot replacing the latest, then passed after serializing renders.
  • .venv/bin/python -m unittest -v -f tests.test_script_runner tests.test_local_metrics passed all 14 tests.
  • .venv/bin/python -m unittest -v -f tests.test_trainer tests.test_validation_external tests.test_training_metrics_service tests.test_training_metrics_routes passed all 132 tests.
  • git diff --cached --check passed.

Closes #3207.

@bghira
bghira merged commit ae0d751 into main Sep 6, 2026
2 checks passed
@bghira
bghira deleted the fix/3207-refresh-upload-report branch September 6, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] training_report.html outdated during upload

1 participant