Skip to content

[None][chore] Add NVTX ranges to per-iteration ADP sync points in PyExecutor#16422

Open
lancelly wants to merge 1 commit into
NVIDIA:mainfrom
lancelly:laliao/nvtx-adp-disagg-sync-ranges
Open

[None][chore] Add NVTX ranges to per-iteration ADP sync points in PyExecutor#16422
lancelly wants to merge 1 commit into
NVIDIA:mainfrom
lancelly:laliao/nvtx-adp-disagg-sync-ranges

Conversation

@lancelly

@lancelly lancelly commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Description

_handle_disagg_cache_errors_synced (a tp_allgather error vote at the top of every executor iteration when disagg + attention DP are enabled) and _can_queue (a post-schedule batch-size tp_allgather under attention DP) are blocking host-side collectives on the executor loop's critical path.

Unlike the sibling disagg status checks (_check_disagg_gen_transfer_status, _check_disagg_ctx_schedulable_status, _check_disagg_*_cache_transfer_status, _pad_attention_dp_dummy_request, _fetch_new_requests), they carry no NVTX annotation, so in nsys traces they show up as anonymous gaps — one at the top of each iteration and one between scheduling and forward — and are easy to misread as scheduler overhead or idle host time.

This PR adds @nvtx_range to both, matching the existing style in the file. No functional change.

Summary by CodeRabbit

  • Performance Monitoring
    • Added runtime tracing markers to improve visibility into executor activity.
    • No user-facing behavior or functionality has changed.

…xecutor

_handle_disagg_cache_errors_synced (tp_allgather error vote at the top of
every executor iteration) and _can_queue (post-schedule batch-size
tp_allgather under attention DP) are blocking host-side collectives, but
unlike the sibling disagg status checks they carry no NVTX range, so they
show up as anonymous gaps in nsys traces. Annotate both for profiling
visibility. No functional change.

Signed-off-by: Liao Lanyu <108499334+lancelly@users.noreply.github.com>
@lancelly
lancelly marked this pull request as ready for review July 15, 2026 10:51
@lancelly
lancelly requested review from a team as code owners July 15, 2026 10:51
@lancelly

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b89ab467-5019-4d4b-a5b9-fccf4c8b5d9e

📥 Commits

Reviewing files that changed from the base of the PR and between e54fe55 and 51011c8.

📒 Files selected for processing (1)
  • tensorrt_llm/_torch/pyexecutor/py_executor.py

📝 Walkthrough

Walkthrough

Adds NVTX range decorators to _can_queue and _handle_disagg_cache_errors_synced in PyExecutor; method logic and control flow remain unchanged.

Changes

PyExecutor instrumentation

Layer / File(s) Summary
Add NVTX ranges
tensorrt_llm/_torch/pyexecutor/py_executor.py
Adds NVTX decorators to _can_queue and _handle_disagg_cache_errors_synced without changing their implementations.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: chienchunhung, 2ez4bz, zheyuf

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description explains the change, but it omits the required Test Coverage and PR Checklist sections from the template. Add the missing Test Coverage and PR Checklist sections, and include any relevant tests or note that none were run.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is specific and matches the NVTX instrumentation change in PyExecutor.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #59441 [ run ] triggered by Bot. Commit: 51011c8 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #59441 [ run ] completed with state SUCCESS. Commit: 51011c8
/LLM/main/L0_MergeRequest_PR pipeline #47909 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@lancelly

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #59899 [ run ] triggered by Bot. Commit: 51011c8 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #59899 [ run ] completed with state FAILURE. Commit: 51011c8
/LLM/main/L0_MergeRequest_PR pipeline #48301 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants