Skip to content

images: mirror digest-pinned indexes byte for byte - #391

Open
mpuig wants to merge 6 commits into
kernel:mainfrom
mpuig:fix/mirror-index-digest
Open

images: mirror digest-pinned indexes byte for byte#391
mpuig wants to merge 6 commits into
kernel:mainfrom
mpuig:fix/mirror-index-digest

Conversation

@mpuig

@mpuig mpuig commented Aug 11, 2026

Copy link
Copy Markdown

Problem

A Dockerfile that pins its base image by digest almost always pins the multi-arch index digest — it's the digest docker pull and Docker Hub print. MirrorBaseImage resolves that reference to the platform image and pushes it under the index digest, which the local registry correctly refuses:

push to local registry: PUT …/v2/library/python/manifests/sha256:9b4929a7…:
unexpected status code 400 Bad Request: digest mismatch:
expected sha256:9b4929a7…, got sha256:1e58d36e…

Because the mirror failure is only a WARN, the build proceeds and fails later with an unrelated error (create builder instance: image is required on v0.3.0), so the root cause is invisible in the API response. Net effect: every digest-pinned FROM is unbuildable — and supply-chain-pinned Dockerfiles are exactly the ones that hit it.

Reproduced on v0.3.0 on a GitHub-hosted ubuntu runner with FROM python:3.13-alpine@sha256:9b4929a7…; journal evidence above.

Fix

An index named by digest is mirrored as the index itself (remote.WriteIndex — children first, then the index manifest), so the destination stores content whose digest is exactly the one the caller pinned.

Unchanged behavior:

  • tag references still mirror only the platform-resolved image (the storage-saving path);
  • a digest reference naming a plain (single-platform) manifest already round-trips and is untouched.

The push logic moves into pushMirrored so the property is testable against in-memory registries (pkg/registry, as in imagepush/manager_test.go): the new tests pin a random two-manifest index by digest, mirror it, and prove the pinned digest resolves at the destination — which is the assertion the old code could not pass.

Testing

  • go test ./lib/images/ green (new + existing tests)
  • go build ./... green

Found while building barista on hypeman; happy to adjust if you'd rather handle index-pinned refs differently (e.g. refusing them loudly instead of mirroring all platforms).


Note

Medium Risk
Changes core base-image mirroring used during builds; behavior splits on digest vs tag refs, with more data copied for digest-pinned indexes but tag/platform mirroring preserved.

Overview
Fixes digest-pinned FROM lines that pin the multi-arch index digest (the usual docker pull output). The mirror used to resolve to a single-platform image and push it under the index digest, which registries reject with digest mismatch; builds then failed downstream with misleading errors.

MirrorBaseImage now fetches via remote.Get and delegates to pushMirrored: digest references that name an index are pushed with remote.WriteIndex so the local registry serves the same digest the Dockerfile pinned. Tag references are unchanged—they still mirror only the platform-resolved image to save storage.

Adds httptest registry coverage for index-by-digest round-trip, tag → single image, and tag + platform selection on a multi-arch index.

Reviewed by Cursor Bugbot for commit 6e08663. Bugbot is set up for automated code reviews on this repo. Configure here.

A Dockerfile that pins its base image by digest almost always pins the
multi-arch *index* digest -- it is the digest `docker pull` and Docker Hub
print. MirrorBaseImage resolved that reference to the platform image and
pushed it under the index digest, which the local registry correctly
refuses:

    push to local registry: PUT .../v2/library/python/manifests/sha256:9b4929a7...:
    unexpected status code 400 Bad Request: digest mismatch:
    expected sha256:9b4929a7..., got sha256:1e58d36e...

The mirror failure is only a WARN, so the build then proceeds and fails
later with an unrelated error -- every digest-pinned FROM is unbuildable,
and supply-chain-pinned Dockerfiles are exactly the ones that hit it.

An index named by digest is now mirrored as the index itself (children
first, then the index manifest), so the destination stores content whose
digest is exactly the one the caller pinned. Tag references keep the
existing platform-image behavior, which is what saves storage; a digest
reference that names a plain manifest is also unchanged.

The push logic moves into pushMirrored so the property is testable against
in-memory registries: the new tests pin a random two-manifest index by
digest, mirror it, and prove the pinned digest resolves at the destination.
mpuig added a commit to mpuig/barista.sh that referenced this pull request Aug 11, 2026
§8 and §9 went upstream as fix PRs (kernel/hypeman#391, #393), §10 and §11
as issues (#394, #395). Each finding now names the workaround it retires
when the upstream fix ships, so the CI hacks have their deletion triggers
on record rather than in memory.
@sjmiller609
sjmiller609 requested a review from chruffins August 11, 2026 21:01

@chruffins chruffins left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for putting up this PR! it's just missing some test coverage but looks good otherwise

Comment thread lib/images/mirror_test.go
mpuig and others added 4 commits August 13, 2026 15:15
…sted platform

Addresses the review note on kernel#391: the existing by-tag test used a single-
platform image, so it proved only that the tag path avoids mirroring the whole
index — not that it resolves a multi-platform index down to the requested
platform. The new test builds a two-platform index (linux/amd64 + linux/arm64),
fetches the tag with WithPlatform(linux/arm64), and asserts the mirror arrives
as that arm64 image: not the index, and not the amd64 sibling.
@mpuig

mpuig commented Aug 23, 2026

Copy link
Copy Markdown
Author

Thanks! Added test coverage in eb23dca8: TestPushMirroredByTagResolvesMultiPlatformIndexToRequestedPlatform builds a two-platform index (linux/amd64 + linux/arm64), fetches the tag with WithPlatform(linux/arm64), and asserts the mirror arrives as that arm64 image — not the whole index, and not the amd64 sibling. The prior single-platform test couldn't distinguish those cases.

@mpuig

mpuig commented Aug 23, 2026

Copy link
Copy Markdown
Author

@chruffins the requested test coverage is in eb23dca8 and the review thread is resolved — CI is green. Could you take another look and re-approve when you get a chance? I can't re-request the review myself from the fork. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants