Skip to content

Support APIC serialization for deterministic CUDA launches #1586

Description

@shi-eric

Description

Follow-up to #1443 and 24ee010.

Deterministic CUDA kernels currently cannot be serialized through APIC. When a deterministic CUDA kernel is launched inside wp.ScopedCapture(..., apic=True), Warp raises a clear RuntimeError explaining that APIC serialization is not currently supported for deterministic CUDA kernels.

This limitation is intentional in the initial deterministic execution work. Deterministic CUDA launch is not a single kernel launch with extra arguments; it may allocate and retain scratch buffers, reset counters and overflow state, launch phase-0 and phase-1 kernels, run native sort/reduce or counter scan/writeback helpers, preserve buffers for CUDA graph replay, and sometimes perform host-side overflow checks.

Desired behavior

APIC capture should either:

  • serialize deterministic CUDA launches using a defined deterministic APIC record format, or
  • expand deterministic launches into APIC-recordable primitives that include the hidden deterministic buffers and native helper calls.

Acceptance criteria

  • Deterministic accumulation kernels can be captured with apic=True, saved, loaded, and replayed correctly.
  • Deterministic consumed-return counter kernels can be captured with apic=True, saved, loaded, and replayed correctly.
  • Hidden deterministic buffers and native helper calls have well-defined lifetime and replay behavior.
  • Unsupported deterministic/APIC combinations continue to fail with clear errors.
  • Tests cover APIC capture rejection or support for both scatter/reduce and counter deterministic paths.

Metadata

Metadata

Assignees

Labels

feature requestRequest for something to be added

Fields

No fields configured for Enhancement.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions