Refocus PR Docker CI on validating the shipped service image

[jioffe502]

Summary

The required PR Docker job provides useful coverage by building the production service image, but its post-build test phase is not focused on the artifact we ship.

After building the image, the workflow installs nemo_retriever[all,dev] into the container and reruns the full non-integration pytest suite. The ordinary unit-test workflow already installs [all,dev] and runs the same test tree. Reinstalling also mutates the service image before testing it, while the job does not start the image's default service command or probe its health endpoint.

Refocus this job on validating the unmodified service image. Keep full container pytest in nightly or release validation.

Current behavior

PR validation:

1. Builds the service target on the pinned Ubuntu base.
2. Starts a container as the image's non-root nemo user.
3. Installs ./nemo_retriever[all,dev] and pytest into the built image.
4. Runs the full nemo_retriever/tests suite with -m "not integration".
5. Uploads coverage.xml, which is not currently consumed by another workflow or coverage gate.

The separate unit-test job already installs nemo_retriever[all,dev] and runs nemo_retriever/tests.

Relevant workflows:

• PR Docker job
• Reusable Docker build-and-test workflow
• Retriever unit-test workflow

Why keep the Docker build

The build has caught real service-image problems. One recent failure showed the service dependency set selecting llvmlite==0.36, which cannot install on Python 3.12. That is the kind of production packaging failure this gate should continue to detect.

Representative run: https://github.com/NVIDIA/NeMo-Retriever/actions/runs/28983968061

Recent observations

For the 100 most recent PR-validation workflow runs inspected on July 14, 2026, covering July 7–14:

• 52 succeeded
• 16 failed
• 30 were canceled, primarily superseded commits
• 2 required approval
• 15 of the 16 failed workflows involved Docker:
  • 13 image-build failures
  • 1 container-test failure
  • 1 Buildx setup failure

These are workflow runs, not distinct product defects; several unrelated documentation branches encountered the same baseline image-build failure.

For the ten most recent successful Docker jobs:

• median duration: approximately 15m41s
• average duration: approximately 15m52s
• range: 14m43s–17m07s

Proposed direction

1. Keep building the production service image in PR validation.
2. Test the image as built, without installing [all,dev] or otherwise changing its dependency set.
3. Add a short artifact-native smoke covering:
   • python -m pip check
   • retriever --help
   • bundled service configuration
   • non-root execution and required writable paths
   • default entrypoint and command
   • service startup followed by a health probe
4. Move full container pytest to scheduled nightly and release validation.
5. Avoid the expensive Docker job for documentation-only changes while keeping branch-protection results deterministic.
6. Add persistent BuildKit caching where appropriate.
7. Either consume the Docker coverage artifact or stop producing it.

Acceptance criteria

• Relevant PRs still build the production service image on Linux amd64.
• Main, nightly, and release workflows retain unconditional image validation.
• PR smoke tests run against the unmodified image dependency set.
• The image runs as its configured non-root user.
• The bundled default entrypoint and command start successfully.
• A health endpoint responds successfully before the smoke test exits.
• PR Docker validation does not reinstall nemo_retriever[all,dev] or rerun the complete unit suite.
• The existing non-container unit suite remains a required PR check.
• Full container pytest runs in nightly or release validation.
• Documentation-only PRs do not incur the full Docker build unless Docker, packaging, service-runtime, or CI inputs also change.
• The coverage artifact is either consumed by a coverage gate or removed.

Non-goals

• Removing Docker validation from CI.
• Changing service runtime behavior.
• Expanding integration tests that require hosted credentials.
• Making this work part of the LanceDB metadata PR.