mac signing: install python 3.14.6 via uv and rebuild venvs when the pin moves - #1342
Draft
hneiva wants to merge 2 commits into
Draft
mac signing: install python 3.14.6 via uv and rebuild venvs when the pin moves#1342hneiva wants to merge 2 commits into
hneiva wants to merge 2 commits into
Conversation
Replaces the python.org framework .pkg (3.11.0) with a uv-managed python-build-standalone build, through a new packages::uvpython class. - Bumps uv 0.8.2 -> 0.12.4. 0.8.2 predates 3.14.6 and does not know how to resolve it; the download list is baked into the uv release. - Drops the `install python3 virtualenv` exec. It pip-installed virtualenv into a hardcoded /Library/Frameworks/.../3.11 path that no longer exists, and nothing consumed it — signing_worker builds its venvs with `uv venv`. - Scopes the `cleanup other UV installs` find to `-name 'uv-*'`. As written it deleted every directory in /opt/tools that was not the current uv, so it would have wiped /opt/tools/python on the next uv bump. - Fails the catalog on non-aarch64, and on macOS older than 14 now that the legacy x86_64 signer branch is gone. packages::uvpython manages the /usr/local/bin links itself instead of using `uv python install --default`. uv's own links resolve through a cpython-<minor>-<platform> symlink that it only ever advances to newer patches: installing 3.14.5 over 3.14.6 leaves `python3 -V` reporting 3.14.6. Pointing at that would put the last-installed patch in charge rather than the pinned $version, and pinning back to an older patch would never satisfy the exec guard, so it would re-run on every apply. Puppet links straight at the patch-specific prefix, which converges in both directions. Note this only takes effect on reprovisioned or new signers. `uv venv` in signing_worker is guarded `onlyif => 'test ! -f .venv/bin/activate'`, so a host with an existing 3.11 venv keeps it and its iscript stays on 3.11. Recreating venvs in place is a separate, more invasive change. Verified: pre-commit clean. `uv python install --no-bin 3.14.6` lands exactly the interpreter path the manifest computes, and is a no-op on a second run. `uv sync --active --locked --inexact --package iscript --extra scriptworker` at the pinned scriptworker_scripts_revision (b87f6ce) resolves and installs on 3.14.6, and iscript, iscript.mac, iscript.script, scriptworker and mozbuild all import — only a non-fatal SyntaxWarning from vendored mozbuild's invalid escape sequence in util.py. Not yet applied to a signer; there is no Kitchen suite covering the mac signing roles.
The venv exec was guarded `onlyif => 'test ! -f .venv/bin/activate'`, so a host with an existing venv kept it indefinitely. Bumping python installed a new interpreter but left every scriptworker running on the old one, which made the 3.14.6 move effective only on reprovisioned hosts. It now compares the venv's python against the pin and rebuilds when they differ. - Pins the version once, in the scriptworker_config defaults anchor, and threads it through mac_signing to both scriptworker_prereqs (the uv-managed interpreter) and signing_worker (the venvs built from it). Two hardcoded copies would drift, and a signing_worker pinning a version that uvpython does not install would rebuild the venv on every apply. - Adds a root `launchctl unload` ahead of the rebuild. The daemon is KeepAlive, so stopping scriptworker any other way just has launchd restart it onto a venv that is being rebuilt underneath it, and $user cannot unload a daemon in the system domain. - `uv venv --clear`, because uv refuses to overwrite an existing venv - the old guard hid that, since it only ever ran when no venv existed. - `--no-python-downloads`, so a pin with no matching local interpreter fails the run rather than silently fetching a private copy into the worker's home. --clear fails before deleting anything, so a bad pin leaves the working venv intact instead of stranding the signer without one. An `rm -rf` first would not have that property. The check reads the version the venv's python reports, which is sound because uv records the resolved patch-specific prefix in the venv rather than the /usr/local/bin/python3 symlink pointing at it. Had it stored the symlink, a bumped link would make a stale venv report the new version while its site-packages stayed old, and the guard would wrongly pass. So one check covers an upgrade, a downgrade, and a base interpreter deleted from under the venv. The restart needs no new wiring: venv -> install iscript -> launchctl_load already chains through refreshonly subscribes. Also leaves a TODO on reprovision_runner's packages::python3, which is unaffected but should be revisited now that the signers no longer use the framework build. Note that a version bump now interrupts running work fleet-wide, since puppet::periodic applies everywhere. launchctl unload sends SIGTERM; whether scriptworker drains its current task or dies with it is unconfirmed, so stage bumps per role if in-flight signing tasks must not be lost. Verified: pre-commit clean. Guard exercised in all four states - matching venv skips, version bump rebuilds, deleted base interpreter rebuilds, absent venv rebuilds. `uv venv --python <pin> --no-python-downloads` resolves an interpreter from PATH without downloading and records the patch-specific prefix; a pin with no local match exits 2 and leaves the existing venv in place. The pin resolves to 3.14.6 for all six signer roles. Not applied to a signer; there is no Kitchen suite covering the mac signing roles.
hneiva
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The signers got python from a python.org framework
.pkgpinned at 3.11.0, while uv was already installed on the same hosts and already building every scriptworker venv (uv venv,uv sync). Python was the last thing left on the old mechanism, and the two didn't agree on anything:packages::python3dropped a framework build in/Library/Frameworks, and a second exec pip-installedvirtualenvinto a hardcoded/Library/Frameworks/.../3.11/bin/python3that nothing consumed.Worse, a version bump couldn't have taken effect anyway. The venv exec was guarded:
Once a venv existed it was never rebuilt. Installing a newer interpreter would have left every scriptworker running on the old one indefinitely, so the pin described the host but not the thing doing the signing.
What changed
New
packages::uvpython. Installs a uv-managed CPython (python-build-standalone) into/opt/tools/pythonand pointspython,python3,python3.14in/usr/local/binat it. uv itself stays where it is; the caller orders this after it.uv 0.8.2 -> 0.12.4. Not cosmetic — uv's download list is baked into the release, and 0.8.2 predates 3.14.6 and cannot resolve it. Checksum verified against the release tarball.
scriptworker_prereqsrestructured into a singlecaseon the macOS version, with an explicit aarch64 guard replacing the "hard-coded to aarch64" comment. The legacy 10.14/10.15 x86_64 branch is gone, so unsupported hosts now fail catalog compilation instead of silently getting a different python.The venv is verified, not merely created. One check drives both new execs:
On mismatch: a root
launchctl unload, thenuv venv --python <pin> --no-python-downloads --clear. The restart needs no new wiring — venv ->install iscript->launchctl_loadalready chains through refreshonly subscribes.The pin lives in one place, the
scriptworker_configdefaults anchor indata/common.yaml, and threads throughmac_signingto both the interpreter install and the venvs built from it. Two hardcoded copies would drift, and asigning_workerpinning a versionuvpythondoesn't install rebuilds the venv on every apply.Four uv behaviours that shaped this
uv python install --defaultwould not have honoured the pin. Its links resolve through acpython-<minor>-<platform>symlink that uv only ever advances to newer patches. Installing 3.14.5 over 3.14.6 leftpython3 -Vreporting 3.14.6. Pointing at that puts the last-installed patch in charge rather than$version, and pinning back to an older patch never satisfies the guard, so the exec re-runs forever. Puppet links straight at the patch-specific prefix instead, which converges in both directions.uv venvresolves the interpreter, and that's what makes the version check sound..venv/bin/pythonpoints at/opt/tools/python/cpython-3.14.6-.../bin/python3.14, not at the mutable/usr/local/bin/python3symlink, andpyvenv.cfgrecordsversion_info = 3.14.6. Had uv stored the symlink, a bumped link would make a stale venv report the new version while itssite-packagesstayed old — the guard would wrongly pass. Because it resolves, one check covers an upgrade, a downgrade, and a base interpreter deleted out from under the venv.uv venvrefuses to overwrite an existing venv — it errors with a--clearhint. The oldonlyifmasked this, since it only ever ran when no venv existed. Any rebuild path needs--clear.--no-python-downloadsis what keeps a bad pin survivable. Without it, a pin uv can't find locally is silently satisfied by downloading a private copy into the worker's home. With it the run fails at exit 2 — and--clearfails before destroying anything, so the working venv is left intact.rm -rf && uv venvwould have deleted it first and then failed, stranding the signer without a venv.Separately, the
cleanup other UV installsfind deleted every directory under/opt/toolsthat wasn't the current uv. It's now scoped to-name 'uv-*'; as written it would have wiped/opt/tools/pythonon the next uv bump.