Refresh deprecating NIMs → 3.5-lightning / nemotron-3-embed-1b@2048 / ultra (eval, proposed) - #129
Open
karthikt-nvidia wants to merge 22 commits into
Open
Refresh deprecating NIMs → 3.5-lightning / nemotron-3-embed-1b@2048 / ultra (eval, proposed)#129karthikt-nvidia wants to merge 22 commits into
karthikt-nvidia wants to merge 22 commits into
Conversation
…bed-1b (2048), eval-judge->ultra Three hosted NIMs on the default cloud path are deprecating (Aug 2026): - LLM (all 4 agents): nemotron-3-nano-30b-a3b -> nemotron-3.5-lightning-30b-a3b - embedder: nv-embedqa-e5-v5 -> nemotron-3-embed-1b (2048-dim; EMBEDDING_DIM made env-driven, Milvus recreated at 2048) - eval judge (nat eval only, proposed): meta/llama-3.1-70b-instruct -> nemotron-3-ultra-550b-a55b Validated end-to-end on a GPU: 14 containers healthy, probe returns real ranked products at dim=2048 with 0 dimension-mismatch errors and a passing negative control. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The post-purchase agent returns a JSON object whose `message` field is a
multi-line shipping email. Without output enforcement the LLM can emit
literal newlines inside that JSON string, which is invalid JSON and fails
the NAT functional check ("post-purchase returned invalid JSON"). It is the
only agent whose output naturally contains newlines, so only this config
needs the guard.
Set response_format: {type: json_object} on the serve LLM. NAT forwards this
to ChatNVIDIA, so the NIM uses guided JSON decoding and emits escaped \n,
guaranteeing a strictly-parseable response.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The recommendation workflow is a sequential ARAG pipeline (NLI scoring ->
user-understanding -> context-summary -> item-ranker) where each chat_completion
stage passes JSON text to the next and output_contract_guard parses it strictly.
If any stage emits malformed JSON the contract guard yields empty
recommendations, so the Apps SDK widget renders zero products ("You May Also
Like" shows nothing) and the dependent cart/checkout UI flows fail.
Set response_format: {type: json_object} on both serve LLMs (nim_llm, nim_llm_nli)
that drive the four chat_completion stages, matching the post-purchase fix. The
search agent is a tool_calling_agent and is deliberately left unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
antoniomtz
self-requested a review
August 19, 2026 23:36
antoniomtz
marked this pull request as ready for review
August 19, 2026 23:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Three hosted NIMs on the default cloud deploy path are deprecating (Aug 2026). This updates all three to their successors, validated end-to-end on a GPU. Every runtime reference is updated (compose + all agent configs); doc/example refs updated for consistency.
nvidia/nemotron-3-nano-30b-a3b→nvidia/nemotron-3.5-lightning-30b-a3bnvidia/nv-embedqa-e5-v5→nvidia/nemotron-3-embed-1b(2048-dim)nat evalonly)meta/llama-3.1-70b-instruct→nvidia/nemotron-3-ultra-550b-a55bThe eval-judge swap is eval-only — it never runs during
nat serve, onlynat eval— and is proposed pending your confirmation; trivial to drop or point elsewhere if you'd prefer a different judge.The one code change: embedding dimension
nemotron-3-embed-1bis 2048-dim (vsnv-embedqa-e5-v5's 1024), andseed_milvus.pyhardcoded the old value. Changed to env-driven (default 2048):The Milvus collection is (re)built at 2048 — a fresh deploy does this automatically; an in-place upgrade needs the vector volume wiped so the collection is recreated at the new width.
Validated on real hardware
Deployed the full cloud-mode stack on a GPU — 14 containers, all healthy. Seeder:
SUCCESS: Seeded 17 products into Milvus,Generated 17 embeddings (dim=2048).Probe (
search-agent /generate, query "running shoes for men"):prod_17 / Athletic Running Shoes(distance 0.9155),count(*) = 17, deployeddim = 2048.Models confirmed from the running containers: all 4 agents on
nemotron-3.5-lightning-30b-a3b; search/recommendation/seeder onnemotron-3-embed-1b;NIM_EMBED_DIM=2048.Note on the new LLM
nemotron-3.5-lightningis a reasoning model — its "thinking" needs to be off for the strict-JSON agent contract, or chain-of-thought leaks into responses and breaks parsing. This repo already shipschat_template_kwargs: enable_thinking: falseon the serve LLM blocks, so no change was needed there; verified 0 chain-of-thought leaks in the logs and clean structured JSON from both a RAG agent and a structured-output agent.Minor (pre-existing, not from this change)
Fail connecting to server on milvus-standalone:19530at boot (depends_ondoesn't gate on Milvus readiness); it retries and goes healthy.Prepared and validated by an automated blueprint-refresh CI (deploy → swap → re-validate on a GPU). Please review before merging.