Skip to content

feat(protocol): add revision 2 multimodal contract and packages#20

Draft
connorcarpenter15 wants to merge 1 commit into
mainfrom
feat/trtllm-multimodal-contract
Draft

feat(protocol): add revision 2 multimodal contract and packages#20
connorcarpenter15 wants to merge 1 commit into
mainfrom
feat/trtllm-multimodal-contract

Conversation

@connorcarpenter15

Copy link
Copy Markdown
Member

Summary

  • advance the OpenEngine schema to revision 2 while retaining minimum client revision 1 compatibility
  • add structured multimodal capability discovery and per-request media options for image, video, and audio
  • preserve the existing multimodal compatibility field and extend generation without removing revision 1 fields
  • add reproducible Rust and Python openengine-proto 0.2.0 package generation, generated-code drift checks, release checks, and cross-language fixtures

Why

The Dynamo TRT-LLM sidecar needs one engine-neutral contract for aggregate generation, context-first prefill/decode handoff, multimodal inputs, LoRA lifecycle, health, load, drain, cancellation, and KV-event discovery. Keeping this contract in OpenEngine lets both consumers pin the same immutable schema release.

Compatibility

Servers advertise schema revision 2 and a minimum client revision of 1. Existing supports_multimodal behavior remains available while newer clients can use the typed modality/source capability fields.

Validation

  • Buf build and lint
  • generated-code drift and descriptor checks
  • Rust binding tests: 4 passed
  • installed Python wheel tests: 6 passed
  • Python package-content and Twine checks
  • Python-to-Rust and Rust-to-Python fixture round trips, including 64-bit identifiers and binary handoff state

Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant