CP full orbit performance: path to ASCOT5 speed (phase "make it fast")
Tracking issue for the deferred performance work on the Cartesian-chartmap full-orbit
CP/CPP (orbit_cpp_boris). We are in phase "make it run": the full orbit is correct
(reactor W7-X high mirror, 1 ms, 1000 alphas: CP 0.896 vs ASCOT5 full orbit 0.898,
GC 0.905 vs 0.904 -- itpplasma/benchmark-orbit-proxima#2). Do NOT optimise yet; this
issue records how to make it fast later, with the analysis and the trick list.
Measured gap (same hardware, acluster, 32 threads, 1000 orbits)
| run |
SIMPLE |
ASCOT5 |
|
| GC 1 ms |
3.0 s |
9.7 s |
SIMPLE 3.5x FASTER |
| GC 100 ms |
188 s |
655 s |
SIMPLE 3.5x FASTER |
| full orbit 1 ms |
409 s |
78 s |
SIMPLE 5.2x SLOWER |
The symplectic GC is faster; the full-orbit CP is ~5x slower. The reason is
architectural, not tuning.
Why the full orbit is ~5x slower
-
SIMPLE inverts, ASCOT reads. ASCOT's field is on a real-space (R,phi,z) grid:
B(x) is one direct tricubic spline read (B_field_eval_B), no inversion. SIMPLE's
field is in logical Boozer/chartmap coordinates while the particle is pushed in
Cartesian, so every B(x) requires solving the Cartesian->logical inverse (a damped
Newton on the chartmap forward map, several spline evals) and then B = curl A.
-
Three inversions per step vs zero. orbit_timestep_cp_boris does, every step:
cpp_boris_step -> 1 cart_field (push), and cpp_boris_to_gc -> 2 locate
calls (orbit_cpp_boris.f90:566,577): one inverts the particle, one inverts the
Larmor-corrected guiding centre. The 2 extra inversions exist because SIMPLE decides
the loss on the reconstructed GC and writes the GC (s,theta,phi) every substep.
ASCOT gets its loss criterion (marker rho>=1) for free from the field read.
-
Finer step. SIMPLE ~38 steps/gyroperiod (npoiper2=16384) vs ASCOT 20
(gyrotime/20): ~1.9x more steps. Both are 2nd-order Boris.
Net: ~3 inversions + 3 field assemblies/step (SIMPLE) vs ~2 direct reads (ASCOT),
x ~1.9 steps, partly offset because ASCOT's 3D tricubic read is itself heavier per
eval than SIMPLE's chartmap spline -> ~5x.
What ASCOT trades for speed (risks of its coarser interpolation)
ASCOT is fast partly by approximations that SIMPLE's curl-A chartmap avoids. The risks
to keep in mind if we copy them:
- Not field-aligned. A (R,phi,z) grid is not aligned to flux surfaces, so it needs
many points to resolve the |B| mirror structure and steep edge gradients; tricubic
error is O(h^4) for B and O(h^2) for grad-B. SIMPLE's chartmap splines in flux
coordinates, where the field varies slowly along the surface, so fewer DOF reach the
same accuracy. A coarse (R,phi,z) grid mis-resolves trapping near the edge/X-point.
- div B != 0. ASCOT splines B_R,B_phi,B_z independently, so the interpolated field
is not exactly divergence-free; SIMPLE's B = curl A is divergence-free by construction.
Non-zero div B can drive spurious parallel acceleration / energy drift over long
(1 s) traces. ASCOT mitigates with fine grids.
- Coarser step. 20 vs 38 steps/gyroperiod: Boris phase error ~ (dt*Omega)^2 grows
faster; over a slowing-down time this is a spurious-radial-transport risk. ASCOT's
volume-preserving VPA bounds the growth; a plain Boris at 20/gyroperiod is riskier.
At ASCOT's validated grid + step the fields agree with our chartmap to 0.01% in |B|
and 0.01 deg in direction (benchmark orbit_cmp), so these are margins, not errors --
but they are the price of the speed, and they shrink at coarser settings.
What VENUS-LEVIS does
Pfefferle et al., CPC 185 (2014) 3127. Full-orbit (and GC) with a Fourier
reconstruction in the poloidal/toroidal angles + cubic spline in the radial flux
coordinate, transformed to Cartesian (PSLINE), RK4. Fourier-in-angle is exact and
periodic (no field-period seam -- cf. our seam bug), flux-aligned (accurate per DOF),
and the interpolation is built to preserve the RK4 integrator order for fast-ion
accuracy. Lesson: a flux-aligned Fourier-spline field is both accurate and seam-free;
the speed comes from an interpolation tuned to the integrator, not from a coarse
real-space grid.
Plan to reach ASCOT speed (ordered by payoff; for "make it fast" phase)
- GC reconstruction off the substep loop. Do
cpp_boris_to_gc only at the
output/loss-check cadence (or only when the particle is within ~a few Larmor radii
of the edge), not every substep. Removes 2 of the 3 inversions/step -- the single
biggest win (~2/3 of per-step cost). Needs a cheap near-edge proxy (the push already
locates the particle, so its rho is in hand for the gate).
- Share the push and to_gc inversions. The push's
cart_field and the first
locate in to_gc invert nearly-adjacent points; warm-seed one from the other or
reuse the result.
- Precompute a direct Cartesian field (ASCOT-style) from the chartmap. Build, once
at setup, a spline of B_cart(x) (and grad|B|, and the loss level set s(x)) on a
Cartesian or (R,phi,z) grid by evaluating the chartmap curl-A field at build time.
The hot loop then reads it directly -- no per-step inversion -- while the chartmap
stays the exact divergence-free source. This is the big architectural lever and
keeps our accuracy advantage.
- Carry the logical coordinate as integration state (ASCOT/GORILLA pattern):
advance u alongside x and source B by the forward map + phi-reduction, eliminating
the inverse in the common case (fall back to inversion only on resync).
- Fewer Newton iterations. Better warm start (predict u from the previous step's
velocity), tighter accept, drop the redundant covariant_basis where evaluate_cart
already returns the Jacobian.
- Vectorise over markers (SIMD). ASCOT pushes NSIMD markers per step; our loop is
one marker per thread. A marker-batched inner loop helps cache + vectorisation.
- Coarsen to ~20 steps/gyroperiod to match ASCOT once accuracy is locked (1 ms
already matches at 38; the finer step buys little).
- Relativistic / volume-preserving push for long-time error control at the coarser
step (matches ASCOT's VPA), removing a reason to keep 38/gyroperiod.
Items 1-2 are local to orbit_cpp_boris/simple.f90 and should recover a large chunk
of the 5x with no physics change; item 3 is the architectural step to actually reach
ASCOT speed; 6-8 close the remainder.
Acceptance (later)
Full-orbit CP within ~1.5-2x of ASCOT5 wall-time at the same loss accuracy
(CP confined within statistics of ASCOT), on the reactor W7-X benchmark.
Related: #419, #420, #421; benchmark itpplasma/benchmark-orbit-proxima#2.
CP full orbit performance: path to ASCOT5 speed (phase "make it fast")
Tracking issue for the deferred performance work on the Cartesian-chartmap full-orbit
CP/CPP (
orbit_cpp_boris). We are in phase "make it run": the full orbit is correct(reactor W7-X high mirror, 1 ms, 1000 alphas: CP 0.896 vs ASCOT5 full orbit 0.898,
GC 0.905 vs 0.904 -- itpplasma/benchmark-orbit-proxima#2). Do NOT optimise yet; this
issue records how to make it fast later, with the analysis and the trick list.
Measured gap (same hardware, acluster, 32 threads, 1000 orbits)
The symplectic GC is faster; the full-orbit CP is ~5x slower. The reason is
architectural, not tuning.
Why the full orbit is ~5x slower
SIMPLE inverts, ASCOT reads. ASCOT's field is on a real-space (R,phi,z) grid:
B(x) is one direct tricubic spline read (
B_field_eval_B), no inversion. SIMPLE'sfield is in logical Boozer/chartmap coordinates while the particle is pushed in
Cartesian, so every B(x) requires solving the Cartesian->logical inverse (a damped
Newton on the chartmap forward map, several spline evals) and then B = curl A.
Three inversions per step vs zero.
orbit_timestep_cp_borisdoes, every step:cpp_boris_step-> 1cart_field(push), andcpp_boris_to_gc-> 2locatecalls (
orbit_cpp_boris.f90:566,577): one inverts the particle, one inverts theLarmor-corrected guiding centre. The 2 extra inversions exist because SIMPLE decides
the loss on the reconstructed GC and writes the GC (s,theta,phi) every substep.
ASCOT gets its loss criterion (marker rho>=1) for free from the field read.
Finer step. SIMPLE ~38 steps/gyroperiod (npoiper2=16384) vs ASCOT 20
(gyrotime/20): ~1.9x more steps. Both are 2nd-order Boris.
Net: ~3 inversions + 3 field assemblies/step (SIMPLE) vs ~2 direct reads (ASCOT),
x ~1.9 steps, partly offset because ASCOT's 3D tricubic read is itself heavier per
eval than SIMPLE's chartmap spline -> ~5x.
What ASCOT trades for speed (risks of its coarser interpolation)
ASCOT is fast partly by approximations that SIMPLE's curl-A chartmap avoids. The risks
to keep in mind if we copy them:
many points to resolve the |B| mirror structure and steep edge gradients; tricubic
error is O(h^4) for B and O(h^2) for grad-B. SIMPLE's chartmap splines in flux
coordinates, where the field varies slowly along the surface, so fewer DOF reach the
same accuracy. A coarse (R,phi,z) grid mis-resolves trapping near the edge/X-point.
is not exactly divergence-free; SIMPLE's B = curl A is divergence-free by construction.
Non-zero div B can drive spurious parallel acceleration / energy drift over long
(1 s) traces. ASCOT mitigates with fine grids.
faster; over a slowing-down time this is a spurious-radial-transport risk. ASCOT's
volume-preserving VPA bounds the growth; a plain Boris at 20/gyroperiod is riskier.
At ASCOT's validated grid + step the fields agree with our chartmap to 0.01% in |B|
and 0.01 deg in direction (benchmark orbit_cmp), so these are margins, not errors --
but they are the price of the speed, and they shrink at coarser settings.
What VENUS-LEVIS does
Pfefferle et al., CPC 185 (2014) 3127. Full-orbit (and GC) with a Fourier
reconstruction in the poloidal/toroidal angles + cubic spline in the radial flux
coordinate, transformed to Cartesian (PSLINE), RK4. Fourier-in-angle is exact and
periodic (no field-period seam -- cf. our seam bug), flux-aligned (accurate per DOF),
and the interpolation is built to preserve the RK4 integrator order for fast-ion
accuracy. Lesson: a flux-aligned Fourier-spline field is both accurate and seam-free;
the speed comes from an interpolation tuned to the integrator, not from a coarse
real-space grid.
Plan to reach ASCOT speed (ordered by payoff; for "make it fast" phase)
cpp_boris_to_gconly at theoutput/loss-check cadence (or only when the particle is within ~a few Larmor radii
of the edge), not every substep. Removes 2 of the 3 inversions/step -- the single
biggest win (~2/3 of per-step cost). Needs a cheap near-edge proxy (the push already
locates the particle, so its rho is in hand for the gate).
cart_fieldand the firstlocateinto_gcinvert nearly-adjacent points; warm-seed one from the other orreuse the result.
at setup, a spline of B_cart(x) (and grad|B|, and the loss level set s(x)) on a
Cartesian or (R,phi,z) grid by evaluating the chartmap curl-A field at build time.
The hot loop then reads it directly -- no per-step inversion -- while the chartmap
stays the exact divergence-free source. This is the big architectural lever and
keeps our accuracy advantage.
advance u alongside x and source B by the forward map + phi-reduction, eliminating
the inverse in the common case (fall back to inversion only on resync).
velocity), tighter accept, drop the redundant covariant_basis where evaluate_cart
already returns the Jacobian.
one marker per thread. A marker-batched inner loop helps cache + vectorisation.
already matches at 38; the finer step buys little).
step (matches ASCOT's VPA), removing a reason to keep 38/gyroperiod.
Items 1-2 are local to
orbit_cpp_boris/simple.f90and should recover a large chunkof the 5x with no physics change; item 3 is the architectural step to actually reach
ASCOT speed; 6-8 close the remainder.
Acceptance (later)
Full-orbit CP within ~1.5-2x of ASCOT5 wall-time at the same loss accuracy
(CP confined within statistics of ASCOT), on the reactor W7-X benchmark.
Related: #419, #420, #421; benchmark itpplasma/benchmark-orbit-proxima#2.