[Tracking] Python toolchain hermeticity: sh_test vs rules_python trade-off

[nicklambourne]

Background

rules_latex deliberately treats python3 as a system utility ("always
on PATH, same as bash") rather than as a managed Bazel dependency.
There is no rules_python dep; the three production-use Python
artifacts (tools/make_cache_snapshot.py,
latex/private/serve_web.py.tpl,
latex/private/serve_watcher.py.tpl) are invoked via
PYTHON="${PYTHON:-python3}" in shell wrappers, and tests in
tests/py/ use sh_test wrapping python3 -m unittest rather than
py_test.

This is documented in-tree in tests/py/BUILD.bazel:

We use sh_test instead of py_test so we don't take a rules_python
dependency for tests - the tests run on the system python3, same as
the snapshot tool itself does in production.

The trade-off is intentional but worth tracking so we can revisit if
the inputs change.

What we get from the current setup

• Zero added Bazel deps. Module footprint stays at platforms +
  bazel_skylib (plus stardoc + rules_shell for dev).
• Symmetric with production: tests exercise the same python3 from $PATH codepath that users actually hit when they bazel run the
  cache snapshot or serve_web target.
• Trivial debug story: python3 -m unittest tests.py.test_foo works
  identically inside and outside Bazel.
• No interpreter download on cold CI cache (~30-100 MB per platform
  saved in repository-cache footprint).
• Faster CI cold-start (~20-40 s saved on first build).

What we lose / where the gaps are

• Not fully hermetic. The sh_test action key doesn't include the
  python3 binary's content hash. bazel test can be a cache hit even
  if the system python3 was upgraded and has a behaviour change. Same
  category of hazard as the synctex-output-group cache poisoning fixed
  in 1978f5d, but slower-moving.
• Stdlib version drift risk. Today we use only conservative stdlib
  APIs that work on 3.9 through 3.13. If we start exercising
  version-sensitive APIs (tomllib, datetime.UTC, PEP 695 generics,
  pathlib.Path.walk, etc.), tests passing locally on 3.13 could fail
  on Ubuntu 22.04's baked-in 3.10.
• No coverage reports. py_test would give us structured test
  output and coverage; sh_test-wrapped unittest gives text only.
• No way to test across Python versions matrix-style without
  external orchestration. With rules_python we could register
  multiple toolchains and test against each.
• Reliance on python3 being present. Users running the full test
  suite need a working Python install. They probably have one (the
  runtime tools need it too), but it's an undocumented prerequisite.

When to revisit

Consider migrating to rules_python + py_test if any of:

1. We start needing version-sensitive stdlib APIs that diverge across
   common Python versions.
2. We need coverage reports (e.g. for confidence in the SyncTeX parser
   under refactor).
3. We want a matrix-test story across Python 3.10/3.11/3.12/3.13
   without spinning up multiple CI runners.
4. The runtime tooling itself grows a third-party dep, which would
   already break the "stdlib-only" promise and remove the symmetry
   argument for keeping tests on system python3.
5. We add a platform to CI where Python isn't reliably installed
   (unlikely for a LaTeX tool, but e.g. minimal container images).

Cost of the migration when we do it

Estimated, not measured:

• rules_python itself: ~2 MB source, fairly stable API as of v1.x.
• Per-platform hermetic interpreter (indygreg
  python-build-standalone): ~30-40 MB compressed,
  ~100 MB unpacked. Materialised lazily, cached in repository-cache.
• CI cold-start: +20-40 s on first build.
• MODULE.bazel changes: add bazel_dep, python extension config,
  use_repo block. Maybe 15 lines.
• Test BUILD files: replace sh_test + run_pytest.sh with py_test
  rules. Maybe 30 lines saved.
• Gap: no Linux arm64 musl prebuilts in python-build-standalone
  (same gap as our biber toolchain, see DESIGN.md §4.9).
• Decide whether runtime tooling stays on system python3 (asymmetric
  with tests) or migrates too (adds runtime dependency on the
  hermetic interpreter, which is then exposed to consumers of
  rules_latex).

Decision record

Status: deferred indefinitely as of v0.2.0. Current setup is the
right trade-off for a stdlib-only LaTeX toolchain. Re-evaluate when
any of the "When to revisit" triggers fires.

See also: tests/py/BUILD.bazel module docstring.