Fix service-mode Compose config drift and NIM startup portability

[jioffe502]

Summary

nemo_retriever/dev/compose/service-mode.compose.yaml still generates the
deprecated local_models.extract.use_graphic_elements setting. The current
ServiceConfig forbids that extra key, so the retriever container enters a
restart loop before the service can start.

The same self-hosted NIM validation exposed three additional portability
problems in the current Compose stack: an unconditional named nvidia runtime
assumption and model-store volume ownership that is incompatible with the
non-root NIM containers, plus repeated OTLP export failures from the embedding
NIM despite telemetry being disabled.

These were discovered while validating #2421 against current upstream/main.
They are Compose/service startup issues, not part of the local OCR batching
change.

Observed failures

1. Deprecated graphic-elements setting prevents service startup

The generated configuration contains:

local_models:
  extract:
    use_graphic_elements: ${LOCAL_EXTRACT_USE_GRAPHIC_ELEMENTS:-true}

Current LocalExtractConfig does not define this deprecated field and uses
extra="forbid". The retriever repeatedly exits with:

local_models.extract.use_graphic_elements
  Extra inputs are not permitted [type=extra_forbidden, input_value=True]

This occurs even with LOCAL_MODELS_ENABLED=false, because the full generated
configuration is validated at startup.

2. Compose assumes a named NVIDIA runtime

The base Compose file sets runtime: nvidia. On the validation host,
docker run --gpus ... and Compose GPU device reservations work, but the
daemon does not register a named nvidia runtime. A temporary override to
runtime: runc retained working GPU reservations and allowed all four core
NIMs to become healthy.

The stack should use one portable GPU-selection mechanism or document and
validate the named-runtime prerequisite.

3. Named model-store volumes are not writable by the NIM users

The page-elements, table-structure, and OCR NIM containers run as non-root
users. Fresh root-owned named volumes mounted at /model-store prevented those
containers from creating their model-store directories. Replacing them with
writable bind mounts allowed startup.

Volume initialization should establish ownership/permissions compatible with
the container user without requiring host-side chmod or ad hoc bind mounts.

4. Disabled telemetry still targets an absent collector

The embedding NIM remained healthy and served requests, but repeatedly logged:

[OTLP HTTP Client] Session state: connection failed.
Could not resolve host: otel-collector

This occurred with NIM_ENABLE_OTEL=false because the Compose environment
still provides TRITON_OTEL_URL=http://otel-collector:4318/v1/traces. The core
NIM-only profile does not start that collector. Disabling telemetry should
also omit the exporter target, or the collector should be included whenever
the target is configured.

Reproduction context

• Source: upstream/main commit 3d9e26f1a2d2fd73af499bb7a9ef7fe855739841
• Command surface: docker compose --profile nims-core -f nemo_retriever/dev/compose/service-mode.compose.yaml up -d
• Page/table image: nvcr.io/nim/nvidia/nemotron-object-detection:2.0.0
• OCR image: nvcr.io/nim/nvidia/nemotron-ocr-v2:2.0.0
• Embedding image: nvcr.io/nim/nvidia/llama-nemotron-embed-vl-1b-v2:1.12.0
• Related experiment record: Batch local OCR crops across page rows #2421

With temporary overrides for the three startup issues, the four NIMs, vector database,
and standalone retriever became healthy, and a 1,360-page ViDoRe service-mode
run completed successfully. The telemetry failure was non-fatal but continued
through a subsequent 19,252-page one-GPU NIM run.

Acceptance criteria

• Remove the deprecated use_graphic_elements environment/config output from the Compose-generated service configuration.
• Add a lightweight validation that the generated Compose service configuration loads through the current ServiceConfig.
• Make GPU runtime selection portable across daemons that support device reservations but do not register a named nvidia runtime, or fail early with a precise documented prerequisite.
• Ensure fresh page-elements, table-structure, and OCR model-store volumes are writable by their container users.
• When telemetry is disabled, do not configure an unreachable OTLP exporter target or emit repeated connection errors.
• Verify the core NIM profile reaches healthy state without a temporary Compose override.
• Verify retriever and vector database reach healthy state in standalone service mode.

Scope

This issue is limited to the self-hosted Compose startup path. It should not
change OCR batching policy, NIM batch-size defaults, #2323 behavior, or
benchmark semantics.