[None][feat] KVCacheManagerV2 C++ translation by lowsfer · Pull Request #14047 · NVIDIA/TensorRT-LLM

lowsfer · 2026-05-12T11:44:10Z

Dev Engineer Review

Review the C++ KVCacheManagerV2 translation, including storage tiers, eviction, radix-tree reuse, CUDA virtual memory, copy/event handling, statistics, lifecycle management, and Python bindings for correctness, performance, API consistency, error handling, and regressions.
Review SHA-256 vendoring, architecture-specific dispatch, CMake integration, pre-commit exclusions, backend selection, and public API compatibility.
Validate configuration checks, memory/resource cleanup, cross-backend behavior, and the MPI shutdown-safety change.
CI follow-up is required: recent pipeline runs failed despite successful helper jobs; rerun with fail-fast disabled, including PerfSanity.

QA Engineer Review

Test code was added or updated in:
- kvCacheManagerV2HostMemTest.cpp
- kvCacheManagerV2StatsTest.cpp
- kvCacheManagerV2TypedIndexTest.cpp
- KV-cache event manager, manager behavior, salting, and stats API Python tests.
Coverage includes host-memory behavior, typed-index contracts, statistics and pending accounting, event parity, reuse introspection, salting, native C++ bindings, and request-ID handling.
No tests/integration/test_lists/, test-db/, or qa/ coverage changes are listed. Test functions should be mapped to CI or manual-QA entries.
Verdict: needs follow-up.

hyukn · 2026-05-15T09:51:10Z

/bot run --disable-fail-fast --stage-list "PerfSanity"

lowsfer · 2026-07-24T06:32:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-24T06:38:37Z

PR_Github #61522 [ run ] triggered by Bot. Commit: bbf42bd Link to invocation

tensorrt-cicd · 2026-07-24T11:11:33Z

PR_Github #61522 [ run ] completed with state FAILURE. Commit: bbf42bd
/LLM/main/L0_MergeRequest_PR pipeline #49738 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-24T11:47:52Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-24T11:53:19Z

PR_Github #61570 [ run ] triggered by Bot. Commit: bbf42bd Link to invocation

tensorrt-cicd · 2026-07-24T18:44:08Z

PR_Github #61570 [ run ] completed with state FAILURE. Commit: bbf42bd
/LLM/main/L0_MergeRequest_PR pipeline #49781 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-27T02:52:24Z

/bot run

tensorrt-cicd · 2026-07-27T02:57:45Z

PR_Github #61783 [ run ] triggered by Bot. Commit: 2cb3efd Link to invocation

tensorrt-cicd · 2026-07-27T04:02:07Z

PR_Github #61783 [ run ] completed with state SUCCESS. Commit: 2cb3efd
/LLM/main/L0_MergeRequest_PR pipeline #49983 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-27T04:26:51Z

/bot run

tensorrt-cicd · 2026-07-27T04:33:25Z

PR_Github #61807 [ run ] triggered by Bot. Commit: 2cb3efd Link to invocation

tensorrt-cicd · 2026-07-27T05:29:14Z

PR_Github #61807 [ run ] completed with state SUCCESS. Commit: 2cb3efd
/LLM/main/L0_MergeRequest_PR pipeline #50006 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-27T06:24:17Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T06:29:45Z

PR_Github #61832 [ run ] triggered by Bot. Commit: 2cb3efd Link to invocation

lowsfer · 2026-07-27T07:20:19Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T07:26:56Z

PR_Github #61848 [ run ] triggered by Bot. Commit: 1fbb70f Link to invocation

tensorrt-cicd · 2026-07-27T07:28:41Z

PR_Github #61832 [ run ] completed with state ABORTED. Commit: 2cb3efd

Link to invocation

tensorrt-cicd · 2026-07-27T08:19:21Z

PR_Github #61848 [ run ] completed with state FAILURE. Commit: 1fbb70f
/LLM/main/L0_MergeRequest_PR pipeline #50042 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Base-branch (main) regressions surfaced by rebasing onto latest main; none are in the KVCacheManagerV2 change set. Two distinct patterns, both fixed by matching the existing sibling conventions. 1) tensorrt_llm import break (nvbugs/6442074): SpecWorkerBase.__init_subclass__ forbids subclasses from overriding forward() -- they must implement _forward_impl so the base forward() wrapper can guarantee spec-dec attn-metadata cleanup. MTPEagleDynamicTreeWorker still overrode forward(), so 'import tensorrt_llm' raised TypeError at class-definition time, breaking all test collection. Rename its forward() to _forward_impl(), matching every sibling worker; callers use the base forward() wrapper so no caller changes. 2) executor tests build objects via __new__ (bypassing __init__) then exercise teardown/lifecycle paths. main NVIDIA#16523 (multi-process HTTP frontends) added attributes to BaseWorker/GenerationExecutorProxy __init__ that teardown reads unguarded, so the __new__-built test objects raised AttributeError: - test_proxy_fast_death.py _bare_proxy: seed _multi_frontend_ipc_dir/_hmac. - test_event_loop_error_broadcast.py _WorkerStub: seed frontend_result_queues (responses_handler read it unguarded -> 4/6 tests failed). - test_proxy_postproc_terminate.py _make_proxy: seed workers_started + _multi_frontend_ipc_dir/_hmac (GC __del__ -> shutdown teardown). - test_base_worker.py __new__ shell: seed doing_shutdown (GC __del__). Each seeds only what that object's teardown reads, matching the _bare_proxy convention. Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

…dings Migrates tensorrt_llm/runtime/kv_cache_manager_v2 from pure Python to a C++ implementation under cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/ with nanobind bindings compiled into bindings.so, preserving the same public API. The dispatcher __init__.py selects the backend via TLLM_KV_CACHE_MANAGER_V2_BACKEND (default "cpp"); both backends pass the shared test suite. CODEOWNERS assigns the new C++ tree to trt-llm-kv-cache-manager-devs. Includes ports of subsequent main features: commit-min-snapshot + SWA-slot reservation, SHA-256 block-key hashing, CUDA-graph request IDs, event manager and stats API to C++, uint64 ReuseScope salt/lora_id, resume-utilization KV constraints, per-conversation KV cache block reuse (PlannedDropHandle), and an MPI teardown fix for the unittest/bindings CI shard. Migration planning docs (TODO.md, MIGRATION_PLAN_CPP.md, CPP_MIGRATION_PLAN_MAIN_15633.md) are kept on a separate branch. Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

lowsfer · 2026-07-27T09:25:39Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T09:31:27Z

PR_Github #61879 [ run ] triggered by Bot. Commit: 0d02e17 Link to invocation

tensorrt-cicd · 2026-07-27T12:56:32Z

PR_Github #61879 [ run ] completed with state FAILURE. Commit: 0d02e17
/LLM/main/L0_MergeRequest_PR pipeline #50072 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-27T13:48:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T13:54:21Z

PR_Github #61916 [ run ] triggered by Bot. Commit: 0d02e17 Link to invocation

tensorrt-cicd · 2026-07-27T16:02:59Z

PR_Github #61916 [ run ] completed with state FAILURE. Commit: 0d02e17
/LLM/main/L0_MergeRequest_PR pipeline #50107 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-27T16:33:14Z

/bot run --disable-fail-fast

lowsfer · 2026-07-27T17:58:43Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T18:04:32Z

PR_Github #61967 [ run ] triggered by Bot. Commit: 0d02e17 Link to invocation

github-actions Bot assigned lowsfer May 12, 2026

lowsfer force-pushed the kvCacheManagerV2-cpp branch 2 times, most recently from 5bd984d to 456cfa8 Compare May 26, 2026 08:44

lowsfer force-pushed the kvCacheManagerV2-cpp branch 3 times, most recently from 4f4dd8d to 0802c6c Compare June 15, 2026 10:25

lowsfer force-pushed the kvCacheManagerV2-cpp branch 6 times, most recently from 4c24d8a to 44c797c Compare June 18, 2026 09:30

lowsfer changed the title ~~Draft: Kv cache manager v2 cpp~~ [None][feat] KVCacheManagerV2 C++ translation Jun 19, 2026

lowsfer force-pushed the kvCacheManagerV2-cpp branch 5 times, most recently from b4592c1 to 5a1fa77 Compare July 10, 2026 10:10

lowsfer added the api-compatible Accepted LLM API contract change that is backwards-compatible label Jul 10, 2026

juney-nvidia requested a review from thorjohnsen July 14, 2026 06:37

lowsfer force-pushed the kvCacheManagerV2-cpp branch 3 times, most recently from fd7d167 to 99bfc50 Compare July 20, 2026 03:03

lowsfer marked this pull request as ready for review July 20, 2026 05:16

lowsfer requested review from a team as code owners July 20, 2026 05:16

lowsfer force-pushed the kvCacheManagerV2-cpp branch from e5a1453 to a4efc5c Compare July 23, 2026 16:13

lowsfer added 2 commits July 27, 2026 09:20

Uh oh!

Conversation

lowsfer commented May 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Uh oh!

hyukn commented May 15, 2026

Uh oh!

lowsfer commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

lowsfer commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

lowsfer commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

lowsfer commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

lowsfer commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

lowsfer commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

lowsfer commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

lowsfer commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

lowsfer commented Jul 27, 2026

Uh oh!

lowsfer commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lowsfer commented May 12, 2026 •

edited by coderabbitai Bot

Loading