Repository navigation
[TRTLLM-17265][refactor] Split openai_protocol into a package - #21544
Draft
JunyiXu-nv wants to merge 17 commits into
Draft
JunyiXu-nv wants to merge 17 commits into
JunyiXu-nv wants to merge 17 commits into
Conversation
…apshots and a CPU test seam for trtllm-serve Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… baselines at the merge base The route reference is generated from the live apps; the object.__new__ baseline and the gRPC import allowance record what exists on main today and only shrink. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the snapshot matrix An attached frontend of a multi-frontend server registers ten launcher forwarding routes before the local table; seven share a method and path with local handlers and three use a path parameter that matches every batch id, so registration order decides which handler serves them. The new configuration records all of it, a live invariant pins the forwarding-before-local order independently of the reference, and the route-class failure message now says why the msgpack route class matters. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
- AGENTS.md: the CPU test command sets LLM_MODELS_ROOT, which the runner requires, and the stage-mapping note says the mapper matches list-entry text, so a file listed through its directory maps by the directory. - check_contracts matches each from-import as <module>.<name>, so 'from pkg import sub' and 'from .. import sub' count as imports of the submodule; new test for both forms and for the allowance entry. - The VisualGen /v1/models test asserts the 200 status. - The AGENTS.md command check also recognizes commands that start with LLM_MODELS_ROOT=. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…lm/serve/_base/msgspec_route.py Generated move: _msgpack_decoder, _MsgspecRequest and _MsgspecRoute leave tensorrt_llm/serve/openai_server.py byte for byte; the module imports _MsgspecRoute back for its app router and keeps no re-export (the three names are private). The two disaggregated test importers now import the new module. OpenAIServer.health, .version and .get_model are removed here and replaced by the system router added in the following commits. tensorrt_llm/serve/_base/msgspec_route.py joins legacy-files.txt (its code comes from a legacy-formatted file), and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ve the system endpoints through it New structure: tensorrt_llm/serve/_base/ (shared serving infrastructure) and tensorrt_llm/serve/api/ (endpoint packages, one router plus one serving class each; the package __init__ imports nothing so routers load lazily, by role). serve/_base/routing.py provides make_router(), which always uses the msgpack-decoding route class, and attach_routers(), which imports and attaches only the routers whose RouterSpec matches the server role. The first endpoint package, serve/api/ops/system/, serves /health, /version and /v1/models for every role. OpenAIServer builds a SystemServing with the injected _health_probe (today's probe body, SIGINT side effect included) and attaches the routers before the role dispatch; the 12 add_api_route lines for the three paths leave the four role tables. Same routes, methods, route classes, bodies and conflict-relevant order: the recorded route reference is unchanged. The layering tests gain the new packages' contracts, the route-snapshot failure message points new endpoints at the router pattern, and the serve agent docs describe where a new endpoint goes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…w serve packages tensorrt_llm/serve/_base/ and tensorrt_llm/serve/api/ops/ join the multi-GPU trigger list: the msgpack route class decodes the disaggregated frontend's worker requests, and /health is the disaggregated readiness probe, so changes under these directories keep running the multi-GPU stages that exercise that path. The three new modules join the strict mypy file set, and the type-check pre-commit hook now also triggers on changes under tensorrt_llm/serve/ and tensorrt_llm/commands/serve_cli/, so edits to typed serving modules get local type checking; the hook still checks only the declared file set. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… tests that build routes by hand Two multi-frontend tests build an OpenAIServer without its constructor and call register_routes() themselves. Since /health, /version and /v1/models are attached through the system router by the constructor, those tests no longer saw the three routes: one asserted that /version stays local on an attached frontend (404), the other reported the three per-frontend-safe entries as removed. Both helpers now set the serving object and attach the routers before the hand registration, in the constructor's order, and the route enumeration walks included routers, which FastAPI 0.139 keeps as objects without a path in app.routes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the OpenEngine metrics test The HTTP-enablement case builds a real OpenAIServer through its constructor and looked for /prometheus/metrics by reading .path off every entry of app.routes. Since the constructor attaches the system router, FastAPI 0.139 keeps an included-router object there that has no .path, and the child process failed with AttributeError. The test now matches a GET /prometheus/metrics request against the routes with the public route-matching API, which works on both ends of the pinned FastAPI range. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…d name The check that endpoint packages under tensorrt_llm/serve/api do not import each other read the package off the from-module alone, so `from tensorrt_llm.serve import api` inside an endpoint package passed and `from tensorrt_llm.serve.api.ops import system` inside ops/system failed. It now reads the package off <module>.<name> per imported name, as the layering contracts do, and a synthetic-edge test pins both forms. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…e/api/visual_gen Move the eight VisualGen request and response models (ImageGenerationRequest, ImageEditRequest, ImageObject, ImageGenerationResponse, MediaReferenceItem, VideoGenerationRequest, VideoJob, VideoJobList) out of tensorrt_llm/serve/openai_protocol.py into the new package tensorrt_llm/serve/api/visual_gen/ (protocol.py), unchanged. openai_protocol keeps the eight names importable through a lazy module __getattr__: importing it no longer loads the VisualGen models or tensorrt_llm.visual_gen, and the re-export is part of its own interface, so it never warns and has no removal release. The imports only the moved block used (UploadFile, AliasChoices, MediaRole) leave openai_protocol with it. Every importer in the repository now imports the new module; the benchmarking guide links the new file. protocol.py joins legacy-files.txt (its code comes from a legacy-formatted file) and the derived lint configs are regenerated. No behavior change. OpenAPI component names may change with the move; they are not a contract. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…isualGen team in CODEOWNERS VisualGen serving code now lives under tensorrt_llm/serve/api/visual_gen/. The new rule, placed after the /tensorrt_llm/serve rule so that it wins, makes @NVIDIA/trt-llm-torch-visual-gen-devs the owner of that directory: changes there need the VisualGen team's approval and no longer need runtime-devs. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…y re-export under a visual_gen path Add tests/unittest/api_stability/visual_gen/test_visual_gen_serve_api.py: it runs the checks of test_serve_api.py on the models of references/trtllm_serve_api.yaml that are defined in tensorrt_llm.serve.api.visual_gen.protocol, and checks that tensorrt_llm.serve.openai_protocol re-exports the eight VisualGen names lazily (importing it does not load the VisualGen module; unknown names still raise AttributeError). The file lives under a visual_gen/ path and has its own l0_cpu.yml entry so that CI selection for VisualGen-only pull requests keeps running it; the reference YAML is unchanged. Add the eight identity rows of the VisualGen names to the VisualGen shim identity table: identity only, no warning and no ban on the old path, because the re-export is openai_protocol's permanent interface. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…its no-warning contract Resolve every pinned model with the full gate's own loader, so a VisualGen model pinned in the reference YAML that is later renamed or removed fails this gate too instead of being dropped silently; the four known models stay a secondary check. Make the lazy-load probe's diagnostic best-effort and identity-based, so a failure prints which modules hold the VisualGen classes instead of crashing on module-level __getattr__ hooks while walking sys.modules. In the same fresh-interpreter probe, assert that reading each of the eight names from openai_protocol emits no FutureWarning or DeprecationWarning. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… one submodule per API tensorrt_llm/serve/openai_protocol.py held every OpenAI-dialect request and response model plus the helpers that turn them into engine parameters. It becomes the package tensorrt_llm/serve/openai_protocol/ with one submodule per API: common, sampling, tools, chat, completions, embeddings, responses, tokenize, models, kv_cache, control and disagg. Every definition moves unchanged (93 definitions; generated by an AST tool and checked definition by definition against the former module). The package __init__.py is a lazy facade: a name-to-module table and a module __getattr__ import a submodule only when one of its names is first read, so every import from tensorrt_llm.serve.openai_protocol keeps working for every name the former module had, the underscore helpers that tests import included, and the VisualGen models re-exported from tensorrt_llm.serve.api.visual_gen.protocol. The facade is the package's own interface, not a compatibility shim: no warning, no removal release, and no importer in the repository changes. Its TYPE_CHECKING block imports the public names from their submodules so type checkers resolve them; the VisualGen names are left out of it on purpose and appear only as strings in the table, so that importing the facade never loads the VisualGen package. DisaggScheduleStyle, which the former module imported and two importers took from it, stays reachable through the facade. Submodules import each other directly, never through the package, along an acyclic graph: common imports no sibling; sampling, tools, models, tokenize, embeddings and control import common; completions and responses import common and sampling; chat imports common, sampling and tools; kv_cache imports common, tools and chat; disagg imports common, completions and chat. torch is imported only by sampling and disagg, openai_harmony only by chat, and the serving-extension registry only by sampling, so code that needs only the tool types no longer loads the chat or Responses models. The 12 submodules are registered in legacy-files.txt (their code comes from a legacy-formatted file) and the deleted module's line is removed; the derived lint configs are regenerated with scripts/legacy_utils.py gen-configs. The facade is new code and stays under the default formatter. No behavior change. OpenAPI component names may change for classes that share a short name with an openai type, because pydantic qualifies them with the module path; they are not a contract. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ziness and layering tests/unittest/serve/openai_protocol/test_openai_protocol_facade.py keeps the package tensorrt_llm/serve/openai_protocol/ honest: every name the former single-file module defined (plus DisaggScheduleStyle) is reachable through the facade and is the object its submodule defines; the facade table is frozen at those names; __all__, dir() and the TYPE_CHECKING block agree with the table; unknown names raise AttributeError; the submodule import graph stays inside the allowed acyclic graph and no submodule imports the package itself; importing the package loads no submodule, importing tools loads neither chat nor responses nor completions, and reading ChatCompletionRequest does not load the serving extensions; the private helpers that other tests import through the package stay importable. tests/unittest/serve/api/visual_gen/test_openai_protocol_facade_visual_gen.py holds the VisualGen assertions under a visual_gen/ path so that CI selection runs them for VisualGen-only changes: the eight VisualGen models map to tensorrt_llm.serve.api.visual_gen.protocol, nothing but a lookup of one of them loads that module, and the facade names it only as strings in its table, never in an import statement. tests/unittest/serve/test_layering.py gains the contract that the protocol package imports no handler module (servers, handlers and routers import the models, never the reverse), and tensorrt_llm/serve/AGENTS.md tells new code to import the submodule that defines a model rather than the package. Both test files are collected by the existing unittest/serve and unittest/serve/api/visual_gen/ entries of l0_cpu.yml and carry the cpu_only marker. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…s as strings and bind the test's culprit list unconditionally The table comment in tensorrt_llm/serve/openai_protocol/__init__.py said that a TYPE_CHECKING import of the VisualGen models would load their package at run time, which a TYPE_CHECKING block never does. The real reason is the static import-isolation check that lets CI run only the VisualGen tests for VisualGen-only changes: it forbids an import edge from this package to the VisualGen package even under TYPE_CHECKING, so the facade names that module only as a string. In tests/unittest/serve/openai_protocol/test_openai_protocol_facade.py the list of submodules that import the serving extensions was bound only on the failing path and read in the assertion message; it is now bound before the branch, so an unconditional message can never raise UnboundLocalError. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #21509 ([TRTLLM-17264][refactor] Move the VisualGen protocol models into serve/api/visual_gen), which is itself stacked on #21448 ([TRTLLM-17261][refactor] Add the trtllm-serve router skeleton and move /health, /version and /v1/models into ops.system) and #21406 ([TRTLLM-17261][test] Add route, middleware-order and shim-identity snapshots and a CPU test seam for trtllm-serve): the first commits below belong to those PRs and disappear from this diff when they merge; this PR's own change is the last 3 commits.
@coderabbitai summary
Description
Turn
tensorrt_llm/serve/openai_protocol.py(2 235 lines: every OpenAI-dialect request and response model oftrtllm-serveplus the helpers that turn them into engine parameters) into the packagetensorrt_llm/serve/openai_protocol/with one submodule per API:common,sampling,tools,chat,completions,embeddings,responses,tokenize,models,kv_cache,control,disagg. Every definition moves unchanged. The package__init__.pyis a lazy facade (a name-to-module table and a module__getattr__) that imports a submodule only when one of its names is first read, so every import fromtensorrt_llm.serve.openai_protocolkeeps working for every name the former module had, underscore helpers included; no importer in the repository changes. After this, a change to one API's models lands in one file of a few hundred lines, long-lived branches stop colliding on one file, and code that needs only the tool types (parsers, for example) no longer loads the chat and Responses models.Part of the trtllm-serve refactor (epic TRTLLM-17056, task TRTLLM-17265: split openai_protocol by API). Builds on #21509, which moved the eight VisualGen models out of the file first (the facade maps their names to
tensorrt_llm.serve.api.visual_gen.protocoland this PR does not move them), on #21448 (theserve/api/package) and on #21406 (the layering test, the shim-identity helpers and theunittest/serveCPU test entries this PR relies on).Behavior
No behavior change. OpenAPI component names may change for classes that share a short name with an
openaitype (pydantic qualifies such names with the module path); they are not a contract.Moves
Pure-move check
The facade is not a compatibility shim: the import path
tensorrt_llm.serve.openai_protocoldoes not change, so it never warns, names no removal release and bans nothing; its identity rows ("no warning, no ban") live in the facade testtests/unittest/serve/openai_protocol/test_openai_protocol_facade.py, not in the shim identity table. The eight VisualGen names are in__all__and in the table but not in the facade'sTYPE_CHECKINGblock: the facade names their module only as strings, so importing it never loads the VisualGen package, and their assertions live undertests/unittest/serve/api/visual_gen/so that CI selection runs them for VisualGen-only changes.History: every submodule shows as a new file (git cannot follow a split);
git blame -C -C tensorrt_llm/serve/openai_protocol/<submodule>.pyattributes the moved lines to their original commits.Import graph between the submodules ("A imports from B"; acyclic):
commonxgrammar,DisaggScheduleStylesamplingtorch, the serving-extension registry, the reasoning-parser factorytools,models,tokenize,embeddings,controlcompletions,responseschatopenai_harmonykv_cachedisaggtorchTwo definitions were added to the former module after the split was planned and are placed by the same rules:
NamespacedFunctionToolParam(aChatCompletionToolsParamsubclass, #21104) goes totools.py;_RESPONSES_HANDOFF_FIELDS(used only byResponsesResponse, #19883) goes toresponses.py.The facade re-exports exactly the former module's definitions plus
DisaggScheduleStyle, which the former module imported and whichserve/openai_disagg_service.pyand its test import through it. The other 70 names the former module imported (AgentHierarchy,Annotated,Any,BaseModel,ChatCompletionAssistantMessageParam,ConfigDict,Dict,Field,GuidedDecodingParams,IncompleteDetails,List,Literal,LlmConversationParams,LlmDisaggregatedParams,LoRARequest,MediaModality,Metadata,NonNegativeInt,OpenAIChatCompletionContentPartParam,OpenAIChatCompletionMessageParam,Optional,PositiveInt,Reasoning,ReasoningEffort,ReasoningParserFactory,Required, the 24openai.types.responsesevent and item classes fromResponseCodeInterpreterCallCodeDeltaEventtoResponseWebSearchCallSearchingEvent,ResponseFormatTextConfig,SamplingParams,Tool,ToolChoice,TypeAlias,TypedDict,Union,base64,check_logprobs_limit,field_validator,importlib,math,model_serializer,model_validator,structured_output_format_for,time,torch,uuid,validate_thinking_token_budget,xgrammar) are not re-exported: they were never part of the protocol, they are importable from their own modules, and pinning them would make every import cleanup inside a submodule a facade change. Nothing in the repository imports any of them throughopenai_protocol.Layering contract added to
tests/unittest/serve/test_layering.py: "openai_protocol imports no handler module" — the package (itsTYPE_CHECKINGimports included) must not importtensorrt_llm.serve.openai_server,openai_disagg_server,openai_disagg_service,openai_service,coordinator_server,openai_video_routes,responses_utils,postprocess_handlers,harmony_adapter,anthropic_adapter,anthropic_batches, nor thetensorrt_llm.serve.api,tensorrt_llm.serve.appandtensorrt_llm.serve.disaggpackages. Servers, handlers and routers import the models, never the reverse.Open PRs that edit
tensorrt_llm/serve/openai_protocol.py(35 on 2026-11-03): a PR that merges before this one is absorbed when the split is regenerated on the latestmainright before merge; a PR that merges after it re-applies its hunk to the submodule below (a rebase stops with a modify/delete conflict on the deleted file).ChatCompletionRequestModelCardDisaggregatedParams,to_llm_disaggregated_paramsChatCompletionRequest,CompletionRequest,ResponsesRequestChatCompletionRequest,CompletionRequestDSparkVerifyLenPinRequestDSparkVerifyLenPinRequest)ChatCompletionResponseChoice,ChatCompletionResponseStreamChoice,DisaggregatedParams,to_llm_disaggregated_paramsChatCompletionRequest,CompletionRequest,ResponsesRequestKvHintParams,KvTransferParams,to_llm_kv_hintKvHintParams,KvTransferParams), disagg.py (to_llm_kv_hint)_logit_bias_to_embedding_biasPromptTokensDetailsRerankDocument,RerankRequest,RerankResponse,RerankResult,RerankUsageInfo,RerankV2ApiVersion,RerankV2BilledUnits,RerankV2Meta,RerankV2Request,RerankV2Response,RerankV2Resultrerank.py(new API; imports only common.py)ChatCompletionRequestRerankDocument,RerankDocumentType,RerankRequest,RerankResponse,RerankResult,RerankUsageInforerank.py(new API; imports only common.py)ResponsesRequest,ResponsesResponse_logit_bias_to_embedding_biasChatCompletionResponseChoiceChatCompletionRequest,CompletionRequest,OpenAIBaseModel,ResponsesRequestChatCompletionRequestFunctionDefinition_KEY_NAMING_ERROR_TYPES,_MAX_LOCATION_SEGMENT_CHARS,_SCALAR_MEMBER_TAGS,_error_location,_is_union_member_tag,_matched_union_member_errors,_union_member_tag_positions,summarize_validation_errors,validation_error_bodyErrorResponse)to_disaggregated_paramsChatCompletionResponseChoiceChatCompletionRequestDisaggregatedParams,to_disaggregated_params,to_llm_disaggregated_paramsDisaggregatedParams,to_disaggregated_params,to_llm_disaggregated_paramsChatCompletionRequest,CompletionRequestChatCompletionRequestResponsesRequestChatCompletionRequestVideoGenerationRequestChatCompletionRequestModelCardChatCompletionRequest_record_sampling_params_request_fieldsChatCompletionRequestOf these, #21155 (Chat Completions: keep replayed tool calls in forwarded requests) adds two
ChatCompletionRequestvalidators that call_materialize_validator_iterators; whichever of the two PRs lands second moves that helper fromresponses.pytocommon.py, so that the chat models do not import the Responses models (the facade test checks the import graph). #21153 (Responses API defaultsparallel_tool_callsto true) editsResponsesRequest; its hunk lands inresponses.pyif it merges second. #19550 / #19624 (Qwen3 reranking) addRerank*models: if one merges first, the regenerated split gives them a new submodulererank.pyimporting onlycommon.py.Deferred on purpose (a move PR changes nothing else): three comments and docstrings still name the former file
openai_protocol.pygenerically, intensorrt_llm/serve/serving_extensions.py(module docstring),tensorrt_llm/scaffolding/worker.py(a comment in the worker) andtests/integration/defs/stress_test/disagg_cancel/harness.py(a docstring), as do two comments undertests/unittest/api_stability/(which also needs the API review committee). The next PR that edits each of those files updates the wording.CI bookkeeping
openai_protocol.pywas not in it; the new package is request/response models, imported by the servers that are)./tensorrt_llm/servecovers the package).tensorrt_llm/serve/openai_protocol/{chat,common,completions,control,disagg,embeddings,kv_cache,models,responses,sampling,tokenize,tools}.py, legacy-formatted group because their code comes from the legacy-formattedopenai_protocol.py) and the deleted module's line removed; derived configs regenerated withscripts/legacy_utils.py gen-configs(ruff-legacy.toml, the managed blocks ofpyproject.tomland.pre-commit-config.yaml);ruff-legacy-baseline.json: none (the old module had no baseline entry). The facade__init__.pyis new code and is not registered.tests/unittest/serve/openai_protocol/test_openai_protocol_facade.pyis collected by theunittest/serveentry ofl0_cpu.yml, andtests/unittest/serve/api/visual_gen/test_openai_protocol_facade_visual_gen.pyby both that entry andunittest/serve/api/visual_gen/(kept for VisualGen-only PRs). Both carrypytest.mark.cpu_only.tensorrt_llm/serve/AGENTS.md: the request/response-model bullet now says to import the submodule, not the package, and names the facade test.Test Coverage
All checks ran on a CPU-only host from this branch's worktree (
PYTHONPATHoverlay over a prebuilt wheel); GPU end-to-end tests are left to CI.pytest -m cpu_only tests/unittest/serve/openai_protocol/test_openai_protocol_facade.py tests/unittest/serve/api/visual_gen/test_openai_protocol_facade_visual_gen.py tests/unittest/serve/test_layering.py→ 21 passed (9 facade cases, 3 VisualGen cases, the layering contracts).pytest tests/unittest/api_stability/test_serve_api.py→ 7 passed, unchanged: the 16 pinned models resolve through the facade; the reference YAML is untouched.pytest -m cpu_only tests/unittest/api_stability/visual_gen/test_visual_gen_serve_api.py tests/unittest/serve/api/visual_gen→ 21 passed (the VisualGen api gate and its lazy re-export cases now run through the facade; the 8 VisualGen identity rows; the routes and ops tests; the new facade test).pytest -m cpu_only tests/unittest/serve→ 124 passed (112 on the base branch plus the 12 new cases).-rsandLLM_MODELS_ROOT→ 79 passed, no "not importable" skip (same count as the base branch).pytest tests/unittest/llmapi/apps/test_serving_extensions.py tests/unittest/llmapi/apps/test_tool_parsers.py tests/unittest/llmapi/test_reasoning_parser.py tests/unittest/llmapi/test_sampling_params.py→ 1495 passed, 1 skipped (a conditional skip intest_reasoning_parser.pythat also skips on the base branch).pytest -m cpu_only tests/unittest/disaggregated→ 1411 passed, 1 skipped, 2 xfailed.pytest -m cpu_only tests/unittest/serve/test_shim_identity.py tests/unittest/llmapi/apps/test_serve_lazy_import.py tests/unittest/serve/test_agents_md.py tests/unittest/serve/test_no_object_new.py→ 26 passed.PYTHONPATH=tests python3 -m test_common.serve.cpu_tests, every serve test-db entry, one pytest process each): 67 entries, 4225 passed, 4 failed, all four in the OpenEngine process tests on a heavily loaded host (load average 16-22, three other test campaigns running): threetest_frontend_lifecycle.py::test_frontend_group_cleanup[...]cases andtest_metrics.py::test_prometheus_contract[multiprocess]. A solo re-run of both files on this branch and on the base branch's worktree in the same time window fails the same threetest_frontend_group_cleanupparametrizations on both sides (3 failed, 18 passed each; which parametrizations exceed their process-wait budget varies between runs with the load) and passes the metrics test. Neither file imports the protocol package.import tensorrt_llm.serve.openai_protocolloads no submodule;[getattr(p, n) for n in p.__all__]resolves all 83 public names; each of the 12 submodules imports on its own and loads exactly its allowed siblings; readingChatCompletionRequestloads neither the serving extensions nor the VisualGen module.AttributeErrorandImportErrorand are absent fromdir(),dir()is the union of the globals and the table, the first read of every name emits no warning,__all__equals the public part of the table and a star-import binds exactly those objects, theTYPE_CHECKINGblock equals__all__minus the VisualGen names → 6 of 6 properties pass; a 10-mutation gate (wrong object returned, unknown name returning None, cache dropped,dir()forgetting the table, a VisualGen row or a private row dropped, a row pointing at the wrong submodule, a warning on first read, a name dropped from__all__or from theTYPE_CHECKINGblock) is detected 10 of 10 (local harness, not committed).pre-commit run --files <every changed file>twice: the second run changes nothing;python scripts/legacy_utils.py check-configspasses;legacy-files.txt,ruff-legacy.tomlandpyproject.tomleach +12/−1 lines,.pre-commit-config.yaml+24/−2 (two managed blocks).git grep -n "openai_protocol\.py"finds only comments and docstrings (the two intests/unittest/api_stability/are left to the next PR that edits those files); no test patches a name on the module; no string anywhere names the module path except the facade's own table and the two api-stability tests.python scripts/test_to_stage_mapping.py --tests "unittest/serve"and--tests "unittest/serve/api/visual_gen/"→CPU-Generic-x86-1/2,CPU-Generic-arm-1/2; both new files containpytest.mark.cpu_only.git blame -C -C tensorrt_llm/serve/openai_protocol/chat.pyattributes 391 of its lines to the commit that last touched them before the split and only the new import lines to this PR.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
I have run the self-review checklist on the PR
🤖 Generated with Claude Code