feat(graph)!: native Dynamo trace replay -- Graph IR agentic workload lane with unified segment store and session routing

[ajcasagrande]

Tip

Q: How does this compare to AIPerf - AgentX?
A: This branch contains more Dynamo specific functaionality than AgentX, but also includes a graph-native alternative implementation of core AgentX concepts, which may be based on this runtime in the future. TBD.

Summary

Native Dynamo trace replay: point AIPerf at a recorded Dynamo capture (.jsonl / .jsonl.gz, segmented trace.NNNNNN.jsonl.gz files, or a directory of them) and replay it faithfully against any OpenAI-compatible endpoint — recorded topology, pacing, token lengths, prefix-cache structure, and session identity included. No conversion step:

aiperf profile \
    --model my-model \
    --url http://localhost:8000 \
    --endpoint-type chat \
    --input-file ./captures/trace.jsonl.gz \
    --streaming \
    --tokenizer builtin \
    --random-seed 1234 \
    --num-dataset-entries 50 \
    --num-conversations 50 \
    --concurrency 8 \
    --concurrency-ramp-duration 60 \
    --workers-max 8 \
    --session-routing dynamo_headers \
    --benchmark-duration 600 \
    --artifact-dir ./artifacts/dynamo-replay \
    --ui simple

The capture is auto-detected as dynamo_trace (pass --graph-format dynamo_trace to force it explicitly). --concurrency-ramp-duration performs a lane-level ramp on the graph replay plane: replay lanes park at phase start and are admitted 1 → --concurrency over the ramp window, spreading load onto a cold server.

Dynamo replay rides a new general agentic-workload lane (Graph IR): LLM workflows represented as dataflow graphs (LLM nodes wired by static edges, reading and writing channels) instead of flat request lists or linear conversations. The same lane also ingests weka_trace, hand-authored native graph YAML/JSONL, and legacy dag_jsonl files.

Dynamo trace replay

• Capture ingestion: dynamo.request.trace.v1 records lower per session-tree (root + descendants linked via agent_context.parent_trajectory_id), so independent trees never share causality edges; schema-less uploader marker lines are tolerated; mixed multi-file captures group cross-file trees correctly.
• Fused parallel build: a hash-free grouping scan routes raw record lines to per-batch workers that read+build in one pass, so the giant recorded hash arrays never cross a process boundary (measured 2.53x on real captures). Corpus-scale memory is bounded by read-time hash-int interning, an offset-based decode cache, segment-id interning, incremental content spill, and a direct write-through store route.
• Content fidelity: recorded block hashes deterministically synthesize prompt content, so equal recorded hashes produce byte-identical blocks — the recorded KV/prefix-cache sharing structure is reproduced and reported via the theoretical prefix-cache metric.
• Replay fidelity: recorded inter-node pacing replays through the shared idle-gap warp (--synthesis-idle-gap-cap), interval-order causality edges enforce recorded finished-before relations, and recorded output lengths pin per-request generation caps.
• Session identity on the wire: --session-routing dynamo_headers stamps X-Dynamo-Session-ID/parent headers; --session-routing dynamo_nvext emits nvext.session_control body metadata, with contract=open matching the released Dynamo v1.2.x contract (open once, then bare session_id) and contract=bind (default) matching >= v1.3.0-dev re-bind-per-turn.

Build plane at a glance

flowchart LR
    subgraph sources["Workload sources"]
        dynamo["dynamo_trace<br/>.jsonl / .jsonl.gz capture"]
        weka["weka_trace<br/>.json / dir / HF corpus"]
        native["native<br/>graph YAML / JSONL"]
        dag["dag_jsonl<br/>legacy DAG files"]
    end

    subgraph ingest["Ingest: aiperf.dataset.graph"]
        ctx["GraphParseContext<br/>run knobs, tri-state idle-gap cap"]
        adapters["graph_adapter registry<br/>parse(path, ctx)"]
        ir["ParsedGraph IR<br/>LlmNodes + edges + channels"]
    end

    subgraph build["GraphStoreBuilder"]
        store["unified segment store<br/>content-addressed, mmap"]
        sidecar["graph_meta sidecar"]
    end

    dynamo --> adapters
    weka --> adapters
    native --> adapters
    dag --> adapters
    ctx --> adapters
    adapters --> ir
    ir --> store
    ir --> sidecar
    store --> bc["DatasetMetadata.graph +<br/>GraphSegmentClientMetadata<br/>broadcast"]
    sidecar --> bc

Loading

Supporting infrastructure (Graph IR lane)

Ingest / IR (aiperf.dataset.graph)

• ParsedGraph schema, parser, structural and semantic validators, and adapters lowering dynamo_trace, weka_trace, native YAML/JSONL, and dag_jsonl onto one IR.
• Node prompts lower into a content-addressed unified segment store keyed by (trace_id, node_ordinal, phase_variant) with a graph_meta sidecar; graph runs broadcast DatasetMetadata.graph + GraphSegmentClientMetadata (mandatory sidecar, no stub conversations).
• Parse dispatch is registry-driven through one GraphParseContext; GraphStoreBuilder owns the store build; trie emission splices content-parent segment chains for corpus-scale CPU bounds.

Runtime (aiperf.graph)

• Executor, scheduler, channel store, credit dispatch adapter, dynamic pools, and worker materialization — graph credits materialize payloads on workers from the unified store, filling dynamic slots from ancestor responses at run time.
• Identity follows the legacy contract: data-inherent {scope}:{turn} node ids, per-trajectory x_correlation_id, instance-keyed sticky sessions with whole-tree co-placement.
• Per-call body/header params are Turn-named native node fields (model, max_tokens, raw_tools, extra_headers, extra_body, theoretical prefix-cache counts).

sequenceDiagram
    participant TM as TimingManager<br/>graph_ir_replay
    participant R as StickyCreditRouter
    participant W as Worker
    participant SR as session_routing plugin
    participant S as Inference server

    TM->>TM: replay recorded pacing<br/>idle-gap warp, t* window
    TM->>R: credit (trace instance, x_correlation_id)
    R->>W: route (instance pinned to ONE worker)
    W->>W: materialize payload from unified store<br/>fill dynamic slots from ancestor responses
    W->>SR: headers() / transform_body() at serialization
    SR-->>W: session identity (headers or nvext body)
    W->>S: HTTP request
    S-->>W: streamed response
    W->>W: capture reply into dynamic pool
    W-->>TM: credit return (unblocks dependent nodes)

Loading

Timing

• graph_ir_replay strategy with a scenario-scoped t* snapshot window, extended-warmup cache-pressure stage with a profiling handoff, and warmup failure aborts.
• Dataset selection (--num-dataset-entries, --max-context-length, --allow-dataset-wrap, sampling strategies) behind a fail-loud wrap-guard; single-pass semantics for bare graph runs.

Session routing

• New session_routing plugin category (--session-routing) unifying router-affinity signaling (dynamo_headers, dynamo_nvext, smg_routing_key, session_id_header) at the request-serialization chokepoint, with per-request lineage/finality facts from SessionTreeRegistry, wired into both the linear and graph planes.
• Behavior change: the exgentic loaders no longer auto-stamp x-dynamo-session-id; pair runs with --session-routing dynamo_headers to restore stamping (now available for any dataset and endpoint).

Fidelity gates

• weka/dynamo cross-format parity (one recording, two encodings, one lowering), dag_jsonl byte-parity vs the legacy plane, golden store digests, and live mock-server E2E runs.

Documentation

User guides (rendered on this branch):

• Agentic Workloads: Native Graph IR — user guide, including the recorded-trace (Dynamo, Weka) quickstart
• Conversation DAG Benchmarks (Legacy) — the predecessor plane and its migration path
• Benchmark Datasets — Exgentic Agent Trace Replay — session-routing behavior change

Reference internals:

• Graph IR schema / validation
• Ingest & build pipeline
• Unified segment store
• Structural handoff
• Async dataflow runtime
• Worker materialization
• Trie prompt convention
• Runtime troubleshooting

🤖 Generated with Claude Code

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@58028d5ba1cb33b0e135e18f1cc8833556a3639e

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@58028d5ba1cb33b0e135e18f1cc8833556a3639e

---

Last updated for commit: 58028d5 • Browse code

[github-actions]

Fern Docs Preview: https://nvidia-preview-9dd1c1ad-9c39-4dd8-b74e-80ece3f77ed2.docs.buildwithfern.com/aiperf/dev

[codecov]

Codecov Report

❌ Patch coverage is 91.00030% with 296 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/aiperf/cli_commands/dynamo_trace_report.py	70.79%	54 Missing and 5 partials ⚠️
src/aiperf/dataset/generator/coding_content.py	67.85%	30 Missing and 6 partials ⚠️
...rf/dataset/graph/adapters/dynamo/trace_parallel.py	89.60%	17 Missing and 12 partials ⚠️
src/aiperf/dataset/generator/_coding_cicd_docs.py	81.93%	25 Missing and 3 partials ⚠️
src/aiperf/common/scenario/validator.py	79.85%	15 Missing and 12 partials ⚠️
src/aiperf/dataset/graph/adapters/dynamo/trace.py	91.73%	8 Missing and 13 partials ⚠️
src/aiperf/cli_runner/_aggregate.py	75.47%	6 Missing and 7 partials ⚠️
src/aiperf/common/scenario/context_overflow.py	76.92%	5 Missing and 4 partials ⚠️
src/aiperf/common/clock.py	92.72%	5 Missing and 3 partials ⚠️
src/aiperf/common/scenario/_env_locks.py	88.40%	4 Missing and 4 partials ⚠️
... and 21 more

📢 Thoughts on this report? Let us know!