Integrate unresolved ARTS2D resonances into detector bins

NOTES.md

@@ -40,3 +40,46 @@ Append-only record of numerical-physics investigations and the decisions needed
 - Across five scan-grid phases, full root-aware quadrature had 0.0393–0.0399% detector L1 error and 0.0569–0.1035% loss-gradient L2 error. The first-order local epsilon/numerator residue correction had 0.0138–0.3162% detector error but 15.5–210.5% gradient error.
 - Corrected CPU medians (100 interleaved blocking calls): full/local warm values were 3.721/3.722 ms and value-plus-gradient calls were 3.805/2.622 ms. Cold value calls were 0.602/0.726 s; cold value-plus-gradient calls were 2.290/2.467 s. There is no useful forward speedup, and the faster local backward pass is numerically wrong.
 - Consequence: implement only the full root-aware quadrature in production. Record the local-correction prototype as a rejected benchmark rather than exposing an inference-unsafe method through the API.
+
+## 2026-08-30 — RESULT: detector-edge and IRF pipeline audit for #134
+
+- ARTS data preparation averages raw wavelength pixels into resolution-unit mean densities. The calibrated wavelength arrays retained by plotting and loss code are bin centers; exact grouped outer edges are now stored separately as `other.detector_specs.electron_wavelength_edges`, including the true width of a ragged final group.
+- The legacy spectral and angular Gaussian samples are now normalized to discrete unit sum, and the hidden post-convolution peak restoration was removed. An unresolved contained delta-line test preserves area to `1e-12` in float64, while its peak is broadened rather than restored.
+- The ATS IRF can now apply angular blur without a second spectral blur when the root-aware quadrature has already returned spectral detector-bin means. Nineteen focused detector/IRF/vectorization tests passed on CPU.
+- Remaining limitation observed during the audit: the calibrated ARTS angle axis is nonuniform (adjacent spacing 0.0771–0.1598 degrees), whereas the legacy angular `jnp.convolve` assumes translation in sample index. Discrete normalization fixes grid-dependent amplitude scaling but does not by itself make angular blur exactly coordinate-aware.
+
+## 2026-08-30 — RESULT: production root-aware detector quadrature for #134
+
+- The production kernel finds sign-changing roots of `Re(epsilon)` on a fixed wavelength scan, differentiates each bisection root implicitly, tan-maps Gauss--Legendre nodes through the local complex-linearized resonance, and integrates exact Gaussian-CDF probability mass into nonuniform detector bins. No collision term or fixed dielectric floor was added.
+- Every gradient/scattering-angle geometry is integrated independently before aperture weighting. A memory-bounded `lax.map` avoids materializing the complete angle-by-node-by-detector tensor. Downstream ARTS processing applies only the angular IRF and angular resolution reduction; it does not convolve or bin wavelength a second time.
+- Fatal numerical conditions (root-capacity overflow, zero local width, non-finite evaluation, or invalid edges/IRF/source bounds/scan phase) poison detector bins with NaN and are also exposed through fixed-shape diagnostics.
+- In the physical 22-degree CPU regression, the 256-panel result varied by at most `2.10e-6` in detector relative L1 and `2.06e-5` in drift-gradient relative L2 across three scan phases. Refining 128 to 256 panels changed values by at most `1.17e-5` and gradients by at most `1.06e-4`, well inside the declared 1% tolerance.
+- Synthetic root benchmarks preserve a constant density and a contained unresolved line's area, support nonuniform detector edges, and match transformed reference values and implicit gradients. Focused production-plumbing tests verify geometry ordering, aperture weighting, calibrated centers/edges, and that no second spectral IRF or spectral reduction occurs.
+
+## 2026-08-30 — RESULT: physical-coordinate spectral and angular IRFs for #134
+
+- This entry supersedes the nonuniform-angular-axis limitation recorded above. On the real 1024-pixel ARTS angular calibration, the legacy index-space convolution gave a centered unresolved line physical area of `1.04699` and shifted its centroid by about `0.084` degrees; its effective physical width also varied with the local pixel spacing. Even-length uniform spectral grids had an additional half-sample centering error.
+- Spectral and angular blur now use an analytic bin-to-bin Gaussian response matrix. Each element exactly integrates the unit-area Gaussian over a finite source cell and output cell on the calibrated physical coordinate, then divides by the output-cell width to return a bin-mean density. This supports even-length and nonuniform axes without peak restoration or an index-spacing approximation.
+- CPU regressions on the real nonuniform angular calibration preserve a centered line's physical area to `2e-11` and centroid to `1e-3` degrees. Uniform 400- and 801-point spectral grids preserve line area to `2e-12` and centroid to `2e-11`; constant densities remain unity away from finite-detector boundaries.
+
+## 2026-08-30 — RESULT: full-range root-scan and refinement validation for #134
+
+- The representative 449–670 nm, 22-degree ARTS2D spectrum has eight sign-changing roots of `Re(epsilon)` at `[474.201875844894, 514.081647830358, 526.276288254028, 526.451996163684, 526.548008175237, 526.723802255372, 539.209067615960, 591.042039725023]` nm. The closest pair is separated by `0.0960120` nm; a root capacity of four was therefore both insufficient and capable of returning a poisoned model over the real detector range.
+- Across 39 scan phases from `-0.95` through `0.95`, 2048 root-scan panels missed a close pair in six phases. Both 4096 and 8192 panels found all eight roots at every phase; their maximum solved-root phase spreads were `1.36e-12` and `6.82e-13` nm. Warm single-geometry medians were `23.5`, `32.2`, and `55.8` ms for 2048, 4096, and 8192 panels. The correctness-first default is therefore 4096 root-scan panels, independent of a 256-panel regular integration grid, with static capacity for 16 roots.
+- A physical narrow-line regression comparing 64 integration panels against 128 and 256 panels at three phases had maximum detector relative-L1 changes of `4.94e-5` and `3.96e-5`; maximum drift-gradient relative changes were `1.86e-4` and `2.64e-4`. The 256-panel phase spreads were `2.10e-6` for values and `2.06e-5` for gradients, all well below the one-percent acceptance threshold.
+- The full-range regression includes phases at which 2048 panels failed, requires exactly the eight roots above, and verifies finite nonzero reverse-mode drift gradients. Its first version exposed NaN cotangents from statically evaluated inactive tan-map candidates; replacing only those inactive evaluations with finite full-panel sentinel mappings fixed the VJP without adding an epsilon floor or changing any active resonance segment.
+- Data-fitting mode carries exact calibrated detector edges, including ragged resolution groups. Forward-only decks have no edge calibration, so `forward_epw_start`/`forward_epw_end` retain their documented legacy role as the first and last wavelength centers; finite outer edges are inferred by half-spacing extrapolation rather than silently reinterpreting those inputs as detector boundaries.
+- The analytic response is stored densely only for the modest nonuniform angular detector axis. Uniform spectral grids use the identical Toeplitz bin-integrated Gaussian kernel, with FFT convolution above 2048 points; this keeps the legacy 10,240-point 1D path memory-linear instead of materializing an approximately 800 MiB float64 response plus intermediates. Dense-versus-Toeplitz regressions agree to `5e-11`, and the 10,240-point 1D forward snapshot remains green.
+
+## 2026-08-30 — RESULT: final #134 validation
+
+- The local-correction comparison and root-scan timings were measured on Apple arm64 with Python 3.14 and JAX 0.9.1. The production conclusion is unchanged: local correction offered no forward speedup and had unacceptable loss-gradient errors, while the full root-aware method remained below the one-percent value/gradient target.
+- The stored ARTS1D golden spectrum was regenerated because it encoded the removed index-space convolution. Replaying the legacy response matched the old snapshot to `5.72e-4` maximum absolute error; the physical-coordinate result differed by `0.104201` at a sharp EPW point. Independent delta-response checks showed that the legacy angular kernel shifted that point by `0.1138` degrees and broadened a requested `0.42466`-degree sigma to `0.5848` degrees, while the analytic response shifted it by `0.0020` degrees and produced `0.4276` degrees after finite-bin integration.
+- Final local CPU validation, with the optional HDF4 dependency installed and MLflow redirected to a file store, completed with `186 passed, 5 skipped` in `196.46` seconds. The five skips are GPU-only tests; the only warning is MLflow's filesystem-backend deprecation. `git diff --check` and Python byte-compilation also passed.
+
+## 2026-08-30 — RESULT: #134 GPU gradient-test audit
+
+- The first manually dispatched stacked-branch workflow exposed two independent runner outcomes: the hosted CPU runner was recycled at 64% after 122 passing tests and one skip (exit 143, no assertion or traceback), while the GPU runner completed the suite and reported that the new scalar-drift detector Jacobian regressions attempted 12.5 GB and 16.0 GB allocations.
+- The GPU allocations came from using reverse-mode `jacrev` for one scalar input and hundreds of detector-bin outputs. This orientation repeats the reverse pass for every output and does not represent the scalar-loss gradient used by inference.
+- The regression now forms the complete scalar-to-detector Jacobian with one forward-mode `jacfwd` tangent evaluation, and independently checks a weighted scalar detector loss with reverse-mode `grad` against the forward-Jacobian contraction. This preserves elementwise tangent coverage and production-style reverse-mode coverage without the artificial quadratic memory cost.
+- The focused CPU file remained green (`3 passed`) and fell from `81.72` to `63.87` seconds; the full-range eight-root test fell from `24.46` to `12.87` seconds.

configs/arts-2d/defaults.yaml

@@ -83,7 +83,7 @@ other:
   calc_sigmas: false
   crop_window: 1
   expandedions: false
-  extraoptions:
+  extraoptions: {}
   flatbg: 0
   gain: 1
   iawfilter:
@@ -99,6 +99,27 @@ other:
   # ~1e-2 in relative L2 on the gradient w.r.t. the EDF -- the gradient is the binding
   # constraint, so raise this if a 2D fit struggles to converge.
   n_beta: 1024
+  # Integrate the collisionless ARTS2D spectrum into finite detector bins before
+  # angular postprocessing. A fine fixed scan finds dielectric roots independently of
+  # the coarser integration grid; root neighborhoods use a tan mapping so resonances much
+  # narrower than a wavelength pixel retain their area and differentiable position. The
+  # spectral Gaussian IRF is included analytically through bin-edge CDF differences and
+  # must not be applied again downstream. Enabled iawfilter edges become exact source-side
+  # integration breakpoints before that IRF is applied.
+  resonance_quadrature:
+    enabled: true
+    root_scan_panels: 4096
+    integration_panels: 256
+    regular_order: 8
+    root_order: 32
+    max_roots: 16
+    neighbor_panels: 1
+    bisection_iterations: 48
+    tail_sigma: 6.0
+    scan_phase: 0.0
+    # Number of gradient/angle spectra evaluated concurrently. One bounds memory most
+    # tightly; raise cautiously on a device with enough memory.
+    map_batch_size: 1
   points_per_pixel: 1
   refit: true
   refit_thresh: 0.25

configs/arts-2d/inputs.yaml

@@ -21,7 +21,7 @@ mlflow:
   experiment: inverse-thomson-scattering
   run: multiplex_arts_rot90_nosmoothtest_nonete_1
 other:
-  PhysParams:
+  detector_specs:
     widIRF:
       ang_FWHM_ele: 1.0
       spect_FWHM_ele: 0.9

docs/source/defaults.rst

@@ -214,9 +214,9 @@ The ``data:`` section contains the specifics on which shot and what region of th
 
     - ``iaw_cf_max`` ending wavelength for a central feature in the IAW that is to be excluded from analysis in nm, must be larger than ``iaw_cf_min``
 
-    - ``forward_epw_start`` starting wavelength in nm for the EPW calculation for forward model only
-    
-    - ``forward_epw_end`` ending wavelength in nm for the EPW calculation for forward model only
+    - ``forward_epw_start`` first wavelength center in nm for the EPW calculation in forward mode. For detector-integrated ARTS2D, the outer detector edge is inferred by half-spacing extrapolation.
+
+    - ``forward_epw_end`` last wavelength center in nm for the EPW calculation in forward mode. For detector-integrated ARTS2D, the outer detector edge is inferred by half-spacing extrapolation.
 
     - ``forward_iaw_start`` starting wavelength in nm for the IAW calculation for forward model only
 
@@ -317,7 +317,9 @@ The ``other:`` section includes options specifying the types of data that are be
 
 - ``gain`` CCD counts per photo-electron; the standard OMEGA ROSS has a gain of 144. Gain must be accurate for appropriate use of Poisson statistics but the gain is generaly not important for the fitting process as the data is normalized by default.
 
-- ``points_per_pixel`` number of wavelength points computed in the spectrum per pixel in the data being analyzed, for most cases 1 is sufficient but if the peaks in the data are very narrow then it may be necessary to use a value larger than 1 to ensure the peaks are well resolved in the computed spectrum, due to the scaling behaviour its not recommended to use a value larger than 10 here.
+- ``points_per_pixel`` number of wavelength points computed per detector pixel by the legacy sampled-spectrum path. ARTS2D does not use this setting to resolve a narrow resonance: it integrates the continuous spectrum directly into the calibrated detector-bin edges.
+
+- ``resonance_quadrature`` controls the ARTS2D detector-bin integration and is enabled by default. ``root_scan_panels`` (4096 by default) is the fine grid used only to bracket sign-changing zeros of the real dielectric, while ``integration_panels`` (256 by default) controls the coarser regular quadrature grid. Separating them resolves closely spaced roots without paying the fine-grid cost in the detector response matrix. ``regular_order`` and the even ``root_order`` set the Gauss--Legendre rules away from and near a root; ``neighbor_panels`` expands the root-mapped neighborhood; ``max_roots`` is the fixed root capacity (16 by default); ``bisection_iterations`` controls the differentiable root solve; and ``tail_sigma`` extends the source integration domain beyond the detector by that many spectral-IRF standard deviations. When ``iawfilter`` is enabled, every filter edge strictly inside that source domain is inserted as an exact integration breakpoint: attenuation is applied to the continuous source spectrum before Gaussian spectral blur and detector integration, including when a filter cuts through a detector bin. ``iawoff`` remains a detector-space mask. ``scan_phase`` is intended for convergence tests and must lie strictly between -1 and 1. ``map_batch_size`` trades device parallelism for peak memory across scattering geometries. A detected root overflow, zero-width resonance, invalid breakpoint, invalid detector geometry, or non-finite evaluation produces a non-finite model rather than silently returning a partial integral.
 
 - ``ang_res_unit`` is the number of pixels in an angular resolution unit for ARTS
 

docs/source/forward_pass.rst

@@ -13,13 +13,14 @@ Other
 
 - ``extraoptions``
 
-    - ``spectype`` the type of spectrum to be computed. This field is self determined from the data when fitting. For a forward pass somthing has to be specified but it deos not effect the spectrum. Options are ``temporal`` or  ``imaging`` In this context they produce the same spectrum.
+    - ``spectype`` the type of spectrum to be computed. This field is determined from the data when fitting, but must be supplied for a forward pass. Options include ``temporal``, ``imaging``, and ``angular``.
 
-    - ``detector_specs`` the subfields define instrumental properties
+- ``detector_specs`` defines instrumental properties and is a sibling of ``extraoptions`` under ``other``.
 
-        - ``widIRF`` the subfields define the instrumental response functions, when fitting data it is determined from the calibration files but when running in forward mode these must be supplied. The subfields are:
+    - ``widIRF`` defines the instrumental response widths. They are read from calibration files when fitting data and must be supplied in forward mode. Its subfields are:
 
-            - ``spect_std_ion`` the standard deviation of the gaussian ion instrumental response function in nanometers
+        - ``spect_stddev_ion`` the standard deviation of the Gaussian ion spectral response in nanometers.
 
-            - ``spect_std_ele`` the standard deviation of the gaussian electron instrumental response function in nanometers
+        - ``spect_stddev_ele`` the standard deviation of the Gaussian electron spectral response in nanometers for non-ARTS spectra.
 
+        - ``spect_FWHM_ele`` and ``ang_FWHM_ele`` the spectral width in nanometers and angular width in degrees, respectively, for ARTS. These two values are full widths at half maximum.

test_angular_2v.py

@@ -62,7 +62,11 @@ def test_arts2d_forward_pass():
             sas = get_scattering_angles(config)
 
             [axisxE, _, _, _, _, _] = get_calibrations(
-                104000, config["other"]["extraoptions"]["spectype"], 0.0, config["other"]["CCDsize"]
+                104000,
+                config["other"]["extraoptions"]["spectype"],
+                0.0,
+                config["other"]["CCDsize"],
+                config["other"]["detector_specs"],
             )  # shot number hardcoded to get calibration
             config["other"]["extraoptions"]["spectype"] = "angular_full"
 

test_arts2v_random.py

@@ -166,7 +166,11 @@ def test_arts2d_inverse(config_path: str = "tests/configs/arts2d_test_defaults.y
             sas = get_scattering_angles(config)
 
             sas["angAxis"], _, _, _, _, _ = get_calibrations(
-                104000, config["other"]["extraoptions"]["spectype"], 0.0, config["other"]["CCDsize"]
+                104000,
+                config["other"]["extraoptions"]["spectype"],
+                0.0,
+                config["other"]["CCDsize"],
+                config["other"]["detector_specs"],
             )  # shot number hardcoded to get calibration
             config["other"]["extraoptions"]["spectype"] = "angular_full"