Skip to content

Full-repo audit and cleanup: correctness, efficiency, redundancy, docstrings, static typing - #132

Merged
almilder merged 2 commits into
mainfrom
pr-4-audit-cleanup
Aug 27, 2026
Merged

almilder merged 2 commits into
mainfrom
pr-4-audit-cleanup

Conversation

@almilder

Copy link
Copy Markdown
Collaborator

Summary

Full-repo audit and cleanup covering correctness, efficiency, redundancy, docstrings, user-facing docs, and static-typing issues surfaced by pyright/Pylance.

Correctness fixes

  • calc_series.py: fixed a real NameError (undefined fe_val/velocity) and a wrong-argument-count call to plot_dist in the non-angular 2D-distribution plotting branch.
  • loss_function.py: loss() called self.unravel_pytree, an attribute that was never assigned (only unravel_weights was ever set); loss_functionals() raised an opaque UnboundLocalError for an unrecognized loss_method instead of a clear ValueError.
  • warpcorr.py: added a clear NotImplementedError for non-EPW instruments instead of a silent crash.
  • distribution_functions/base.py: calc_moment now raises a clear error for unsupported distribution-function shapes instead of returning an unbound variable.
  • load_ts_data.py: fixed two invalid \m escape sequences in axis-label strings.

Efficiency improvements

  • loss_function.py: removed a duplicate forward-model call in calc_loss, vectorized calculate_covariance_matrix with vmap instead of an unrolled Python loop.
  • form_factor.py: cached the Z-prime table disk reads instead of reloading them on every FormFactor construction.
  • postprocess.py / calc_series.py: hoisted ThomsonScatteringDiagnostic construction out of per-batch/per-iteration loops.
  • warpcorr.py: vectorized the pixel-by-pixel dewarping loop (up to ~1M iterations per shot) using np.add.at-based scatter-accumulation; verified byte-exact against the original loop-based implementation on real shot data.
  • correct_throughput.py: replaced a tile+transpose allocation with plain broadcasting, and a cell-by-cell xlrd read loop with vectorized col_values calls; also drops the deprecated numpy.matlib.repmat import. Both verified exact-match against the originals.

Redundancy / simplification

  • generate_spectra.py: consolidated ion_spectrum/electron_spectrum and their _detailed counterparts around shared core methods; verified bit-exact against the pristine pre-refactor implementation on both the fast and detailed paths.
  • plotters.py: factored detailed_lineouts's duplicated worst/best plotting blocks into a shared helper, fixing a real bug uncovered in the process (the "worst" block never set the ion subplot's y-limit due to a copy-paste slip); factored the duplicated white-transition colormap construction.
  • example_plot.py: deleted -- never functional as written, and nothing referenced it beyond a stale auto-generated docs stub.
  • edf_movie.py: replaced a hardcoded personal file path with a documented, editable constant.

Docstrings

Added missing public function/method docstrings across the touched modules, and module-level docstrings to every file in the package that lacked one.

Documentation

docs/source/getting_started.rst now documents the standalone postprocessor runner and cluster job submission (queue_tsadar.py, the machine config field, and the required Slurm environment variables), neither of which were previously documented.

Test plan

@almilder
almilder changed the base branch from main to pr-3-oom-fix August 20, 2026 19:18
@almilder
almilder force-pushed the pr-4-audit-cleanup branch from 5557786 to 9a9ae8e Compare August 25, 2026 14:44
@almilder
almilder force-pushed the pr-4-audit-cleanup branch from 9a9ae8e to 4e5aa93 Compare August 26, 2026 14:44
@almilder
almilder force-pushed the pr-4-audit-cleanup branch from 4e5aa93 to b60c887 Compare August 26, 2026 17:12
Base automatically changed from pr-3-oom-fix to main August 27, 2026 13:45
…strings, static typing

Correctness fixes:
- calc_series.py: fixed a real NameError (undefined fe_val/velocity) and a
  wrong-argument-count call to plot_dist in the non-angular 2D-distribution
  plotting branch.
- loss_function.py: loss() called self.unravel_pytree, an attribute that was never
  assigned (only unravel_weights was ever set); loss_functionals() raised an opaque
  UnboundLocalError for an unrecognized loss_method instead of a clear ValueError.
- warpcorr.py: added a clear NotImplementedError for non-EPW instruments instead of
  a silent crash.
- distribution_functions/base.py: calc_moment now raises a clear error for
  unsupported distribution-function shapes instead of returning an unbound variable.
- load_ts_data.py: fixed two invalid \m escape sequences in axis-label strings.

Efficiency improvements:
- loss_function.py: removed a duplicate forward-model call in calc_loss, vectorized
  calculate_covariance_matrix with vmap instead of an unrolled Python loop.
- form_factor.py: cached the Z-prime table disk reads instead of reloading them on
  every FormFactor construction.
- postprocess.py / calc_series.py: hoisted ThomsonScatteringDiagnostic construction
  out of per-batch/per-iteration loops.
- warpcorr.py: vectorized the pixel-by-pixel dewarping loop (up to ~1M iterations per
  shot) using np.add.at-based scatter-accumulation; verified byte-exact against the
  original loop-based implementation on real shot data.
- correct_throughput.py: replaced a tile+transpose allocation with plain broadcasting,
  and a cell-by-cell xlrd read loop with vectorized col_values calls; also drops the
  deprecated numpy.matlib.repmat import. Both verified exact-match against the
  originals.

Redundancy / simplification:
- generate_spectra.py: consolidated ion_spectrum/electron_spectrum and their _detailed
  counterparts around shared core methods; verified bit-exact against the pristine
  pre-refactor implementation on both the fast and detailed paths.
- plotters.py: factored detailed_lineouts's duplicated worst/best plotting blocks into
  a shared helper, fixing a real bug uncovered in the process (the "worst" block never
  set the ion subplot's y-limit due to a copy-paste slip); factored the duplicated
  white-transition colormap construction.
- example_plot.py: deleted -- never functional as written, and nothing referenced it
  beyond a stale auto-generated docs stub.
- edf_movie.py: replaced a hardcoded personal file path with a documented, editable
  constant.

Docstrings: added missing public function/method docstrings across the touched
modules, and module-level docstrings to every file in the package that lacked one.

Documentation: docs/source/getting_started.rst now documents the standalone
postprocessor runner and cluster job submission (queue_tsadar.py, the machine config
field, and the required Slurm environment variables), neither of which were
previously documented.

Full test suite (pytest tests/) passes: 125 passed, 5 skipped, throughout.
@almilder
almilder force-pushed the pr-4-audit-cleanup branch from b60c887 to b293e66 Compare August 27, 2026 13:45
launch_data_visualizer gated the background-lineout plot on config["data"]
["background"]["report_background"] with a bare dict lookup, so any config
predating that field (an external deck, or one from before this PR) would
raise an unhelpful bare KeyError: 'report_background' from inside
prepare_data. Checks for the key explicitly first and raises with a message
that says what's missing and how to fix it, matching the pattern already
used by the calibration fallback's missing-key errors.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@almilder

Copy link
Copy Markdown
Collaborator Author

Review summary

Reviewed the full diff against main. The bulk of this PR is docstrings, comments, and faithful refactors — deduplicating near-identical code in plotters.py/generate_spectra.py, vectorizing warpcorr.py's dewarp loop, and caching zprimeMaxw's lookup tables. Every substantive behavioral change was traced against its pre-PR equivalent and found to be equivalent or a genuine fix, including two incidental correctness fixes bundled in:

  • unravel_pytree → unravel_weights fixes a real AttributeError that exists on main today.
  • log_mlflow's % → ceiling-division fix corrects real silent data-dropping for batches of >1000 entries.
  • The plot_dist call-site fix removes a stale extra positional arg that would have raised TypeError.

Finding

  • tsadar/data/data_visualizer.py — the lineouts-with-background plot was gated by a bare config["data"]["background"]["report_background"] lookup with no default, replacing the old type in ["fit", "pixel"] check that only ever relied on type (a key every config has always had). Every config shipped in this repo has report_background now (added in Close matplotlib figures in the data visualizer to prevent an OOM error #131), but any external or hand-maintained deck that predates that change has no way to know it needs the key, and would hit a bare KeyError: 'report_background' from inside prepare_data on a fresh fit with launch_data_visualizer: true — where the same deck previously ran fine.

Fix

Checks for the key explicitly and raises with a message that says what's missing and how to fix it, instead of a bare KeyError:

if "report_background" not in config["data"]["background"]:
    raise KeyError(
        "config['data']['background'] is missing 'report_background'; add it (True/False) to your deck"
    )

@almilder
almilder merged commit acfa52c into main Aug 27, 2026
2 checks passed
@almilder
almilder deleted the pr-4-audit-cleanup branch August 27, 2026 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant