Skip to content

CP full orbit performance: path to ASCOT5 speed (phase 'make it fast') #422

Description

@krystophny

CP full orbit performance: path to ASCOT5 speed (phase "make it fast")

Tracking issue for the deferred performance work on the Cartesian-chartmap full-orbit
CP/CPP (orbit_cpp_boris). We are in phase "make it run": the full orbit is correct
(reactor W7-X high mirror, 1 ms, 1000 alphas: CP 0.896 vs ASCOT5 full orbit 0.898,
GC 0.905 vs 0.904 -- itpplasma/benchmark-orbit-proxima#2). Do NOT optimise yet; this
issue records how to make it fast later, with the analysis and the trick list.

Measured gap (same hardware, acluster, 32 threads, 1000 orbits)

run SIMPLE ASCOT5
GC 1 ms 3.0 s 9.7 s SIMPLE 3.5x FASTER
GC 100 ms 188 s 655 s SIMPLE 3.5x FASTER
full orbit 1 ms 409 s 78 s SIMPLE 5.2x SLOWER

The symplectic GC is faster; the full-orbit CP is ~5x slower. The reason is
architectural, not tuning.

Why the full orbit is ~5x slower

  1. SIMPLE inverts, ASCOT reads. ASCOT's field is on a real-space (R,phi,z) grid:
    B(x) is one direct tricubic spline read (B_field_eval_B), no inversion. SIMPLE's
    field is in logical Boozer/chartmap coordinates while the particle is pushed in
    Cartesian, so every B(x) requires solving the Cartesian->logical inverse (a damped
    Newton on the chartmap forward map, several spline evals) and then B = curl A.

  2. Three inversions per step vs zero. orbit_timestep_cp_boris does, every step:
    cpp_boris_step -> 1 cart_field (push), and cpp_boris_to_gc -> 2 locate
    calls (orbit_cpp_boris.f90:566,577): one inverts the particle, one inverts the
    Larmor-corrected guiding centre. The 2 extra inversions exist because SIMPLE decides
    the loss on the reconstructed GC and writes the GC (s,theta,phi) every substep.
    ASCOT gets its loss criterion (marker rho>=1) for free from the field read.

  3. Finer step. SIMPLE ~38 steps/gyroperiod (npoiper2=16384) vs ASCOT 20
    (gyrotime/20): ~1.9x more steps. Both are 2nd-order Boris.

Net: ~3 inversions + 3 field assemblies/step (SIMPLE) vs ~2 direct reads (ASCOT),
x ~1.9 steps, partly offset because ASCOT's 3D tricubic read is itself heavier per
eval than SIMPLE's chartmap spline -> ~5x.

What ASCOT trades for speed (risks of its coarser interpolation)

ASCOT is fast partly by approximations that SIMPLE's curl-A chartmap avoids. The risks
to keep in mind if we copy them:

  • Not field-aligned. A (R,phi,z) grid is not aligned to flux surfaces, so it needs
    many points to resolve the |B| mirror structure and steep edge gradients; tricubic
    error is O(h^4) for B and O(h^2) for grad-B. SIMPLE's chartmap splines in flux
    coordinates, where the field varies slowly along the surface, so fewer DOF reach the
    same accuracy. A coarse (R,phi,z) grid mis-resolves trapping near the edge/X-point.
  • div B != 0. ASCOT splines B_R,B_phi,B_z independently, so the interpolated field
    is not exactly divergence-free; SIMPLE's B = curl A is divergence-free by construction.
    Non-zero div B can drive spurious parallel acceleration / energy drift over long
    (1 s) traces. ASCOT mitigates with fine grids.
  • Coarser step. 20 vs 38 steps/gyroperiod: Boris phase error ~ (dt*Omega)^2 grows
    faster; over a slowing-down time this is a spurious-radial-transport risk. ASCOT's
    volume-preserving VPA bounds the growth; a plain Boris at 20/gyroperiod is riskier.

At ASCOT's validated grid + step the fields agree with our chartmap to 0.01% in |B|
and 0.01 deg in direction (benchmark orbit_cmp), so these are margins, not errors --
but they are the price of the speed, and they shrink at coarser settings.

What VENUS-LEVIS does

Pfefferle et al., CPC 185 (2014) 3127. Full-orbit (and GC) with a Fourier
reconstruction in the poloidal/toroidal angles + cubic spline in the radial flux
coordinate
, transformed to Cartesian (PSLINE), RK4. Fourier-in-angle is exact and
periodic (no field-period seam -- cf. our seam bug), flux-aligned (accurate per DOF),
and the interpolation is built to preserve the RK4 integrator order for fast-ion
accuracy. Lesson: a flux-aligned Fourier-spline field is both accurate and seam-free;
the speed comes from an interpolation tuned to the integrator, not from a coarse
real-space grid.

Plan to reach ASCOT speed (ordered by payoff; for "make it fast" phase)

  1. GC reconstruction off the substep loop. Do cpp_boris_to_gc only at the
    output/loss-check cadence (or only when the particle is within ~a few Larmor radii
    of the edge), not every substep. Removes 2 of the 3 inversions/step -- the single
    biggest win (~2/3 of per-step cost). Needs a cheap near-edge proxy (the push already
    locates the particle, so its rho is in hand for the gate).
  2. Share the push and to_gc inversions. The push's cart_field and the first
    locate in to_gc invert nearly-adjacent points; warm-seed one from the other or
    reuse the result.
  3. Precompute a direct Cartesian field (ASCOT-style) from the chartmap. Build, once
    at setup, a spline of B_cart(x) (and grad|B|, and the loss level set s(x)) on a
    Cartesian or (R,phi,z) grid by evaluating the chartmap curl-A field at build time.
    The hot loop then reads it directly -- no per-step inversion -- while the chartmap
    stays the exact divergence-free source. This is the big architectural lever and
    keeps our accuracy advantage.
  4. Carry the logical coordinate as integration state (ASCOT/GORILLA pattern):
    advance u alongside x and source B by the forward map + phi-reduction, eliminating
    the inverse in the common case (fall back to inversion only on resync).
  5. Fewer Newton iterations. Better warm start (predict u from the previous step's
    velocity), tighter accept, drop the redundant covariant_basis where evaluate_cart
    already returns the Jacobian.
  6. Vectorise over markers (SIMD). ASCOT pushes NSIMD markers per step; our loop is
    one marker per thread. A marker-batched inner loop helps cache + vectorisation.
  7. Coarsen to ~20 steps/gyroperiod to match ASCOT once accuracy is locked (1 ms
    already matches at 38; the finer step buys little).
  8. Relativistic / volume-preserving push for long-time error control at the coarser
    step (matches ASCOT's VPA), removing a reason to keep 38/gyroperiod.

Items 1-2 are local to orbit_cpp_boris/simple.f90 and should recover a large chunk
of the 5x with no physics change; item 3 is the architectural step to actually reach
ASCOT speed; 6-8 close the remainder.

Acceptance (later)

Full-orbit CP within ~1.5-2x of ASCOT5 wall-time at the same loss accuracy
(CP confined within statistics of ASCOT), on the reactor W7-X benchmark.

Related: #419, #420, #421; benchmark itpplasma/benchmark-orbit-proxima#2.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions