Build reproducibility for automated release signing

[mengw15]

Background

While moving our release signing to a policy-compliant setup (see PJ's mail on
dev@, "signing key used for create-release-candidate.yml"), one precondition of
the ASF automated-signing policy is that all signed artifacts can be built
reproducibly — bit-for-bit — so anyone can independently rebuild them and
byte-compare against what CI produced. That is what makes a machine's signature
auditable: if what you built and what CI built are identical, the signature
attests to verifiable bytes.
(Policy: https://infra.apache.org/release-signing.html#automated-release-signing)

Where Texera stands

Good news first: both voted artifacts (the source tarball and the
docker-compose bundle) are repackagings of committed files — nothing is
compiled and no dependencies are resolved at build time — so this is far
easier for us than for projects shipping binaries. (Transitive dependencies
only come into play for the docker images, which are convenience binaries
outside the signed set.)

Today the artifacts are not reproducible, purely for packaging-metadata
reasons:

• plain tar -czf picks up filesystem file ordering and the runner's uid/gid,
• gzip embeds a compression timestamp,
• the compose bundle's processed files (e.g. the generated .env) carry fresh
  mtimes.

Demonstration — same commit (release/v1.3 tip), the current packaging run
twice:

1d07e1728727…   first run
7706218a6fd3…   second run   <- different bytes

versus a one-step git archive --format=tar.gz, run twice:

4c7851fe0e03…   first run
4c7851fe0e03…   second run   <- identical

Proposal

This looks fixable with a small, contained change:

1. Source tarball: produce it in one step with git archive (deterministic by
   construction — git tree order, commit-time mtimes, no gzip timestamp).
2. Compose bundle: deterministic GNU tar flags
   (--sort=name --owner=0 --group=0 --mtime=@<commit-ts>) plus gzip -n.
3. A verify-reproducible script so anyone can rebuild both artifacts from a
   tag and byte-compare against the staged ones — this becomes the policy's
   "validate on trusted hardware" step before promotion.
4. Optionally a CI job that builds everything twice and fails on any byte
   difference, so reproducibility stays enforced.

Scope note: this covers the voted artifacts. Docker images are convenience
binaries outside the signed set; making them reproducible is a much bigger
topic and not required by the policy for our case.

Happy to put up the PR if this sounds right — feedback welcome.

[parshimers]

I think this follows even for the docker images, or maybe especially for them. So, it isn't just constrained to the source tarball and compose files. Rather, it should be that you can produce absolutely everything from that source tarball.
https://reproducible-builds.org/docs/jvm/ has some good reference regarding this. It seems sbt has some tooling available already.
Another important thing, is that for release branches (and honestly I think for main as well), everything external that is depended on needs to be at a version. It can't be latest, a nightly, or a snapshot.
However I'm less sure about NPM and the JS ecosystem in general. If someone with more expertise in that area could chime in about that, it would be appreciated.

[mengw15]

Thanks Ian, good point on the images.

Two questions:

1. For the signing precondition, is it enough that the artifacts we actually sign and vote on — the source tarball and the compose bundle — are bit-reproducible? They reference images but don't contain them, and the images are convenience binaries outside the signed set.

2. Who decides that for us? Is it something you or the IPMC would weigh in on, or does it get settled when we write to security@ for the CI signing key?

[parshimers]

I suppose it all lies in how you interpret this, from the automated signing documentation:

The release process contains a validation step where all artifacts are reproduced and validated on trusted hardware (i.e. not GitHub Actions) before publication to pages intended for end users. This means rebuilding the artifacts from source and validating the resulting artifacts are bit-by-bit identical to those staged for release.

The way I interpret that is: Whatever is released, signed or not, has to be bit-for-bit reproducible if you want to release it through hardware that isn't owned by you, personally, as a committer. Otherwise there's no way to tell if something has been done to the artifact to compromise it on this hardware that you don't have full control over.

[parshimers]

I opened a draft PR for this, it's just a take at getting Claude to do it: #8812. To be completely clear: I'm not doing this because it's somehow a Mentor's job to do it, or that you might expect a Mentor to handle things like this. Quite to the contrary, it is the incubating project's burden to deal with making releases compliant with ASF policies. I'm just doing it (as a contributor) because I think it's an interesting problem that I hadn't fully grasped before, and one I will also have to solve in AsterixDB. So, Texera can be the guinea pig ;) I would also appreciate any help with it, if someone likewise finds the topic interesting.

Some of the things I don't quite understand yet, like how purportedly an image which installs something using a package manager is indeed deterministic. I can see how that could be true, but I can also see how it wouldn't be true.

[parshimers]

OK, I looked a bit more at #8812 and I understand it somewhat now.
The very clever solution Claude pulled out here is to use the ubuntu/debian snapshot archives, and date gating on PyPI, to basically guarantee a set of packages that go into one of our dockerfiles can be deterministic. I was thinking version pinning would be the way to go about this but apparently the packages can change sometimes even if the versions are the same.
I think it needs some more manual testing though. I also still don't entirely get why none of the things I would have thought would be necessary for the Scala part of the build seem to be. I guess somehow we don't run into some of the usual things that make jars non-deterministic from source, but I still would like to fully understand how.

[mengw15]

Results from my side (Apple Silicon host; BuildKit v0.33.1, GNU tar 1.35). Commit 9ac8c5d5a, SOURCE_DATE_EPOCH=1791491650 (commit time), PACKAGE_SNAPSHOT=2026-10-08, --platform linux/amd64, built exactly per bin/README.md:

bash .github/scripts/create-release-tarballs.sh 9ac8c5d5a4ef6389b546b344f0a91209f26a55f5 1.3.0-reprotest docker.io/apache 1.3.0-reprotest out

docker buildx build --platform linux/amd64 \
  --build-arg SOURCE_DATE_EPOCH --build-arg PACKAGE_SNAPSHOT=2026-10-08 \
  --output type=oci,dest=<name>.tar,rewrite-timestamp=true \
  -f bin/dockerfiles/<name>.dockerfile .

Everything is deterministic locally (tarballs and a re-built image byte-identical across repeat runs; jOOQ = 223 files, matching yours). Tarballs:

486ecd7606ada49d8c57c6691832ef83228f5f332d5a033aa2e365489c21d77a4edd4b0b1c5e6df7475423be0fad1cca610a1b6ac5b885dd371dc6afe0f36d9a  apache-texera-1.3.0-reprotest-src.tar.gz
bdc5882b36675bd0df68c4deb45e7b97739de4ccdd8c88e293a8dfb9b7f50aea24aa7d55ab26b2c5a1d057e784d4721fcb53d96618ae3906048fef01edb397df  apache-texera-1.3.0-reprotest-docker-compose.tar.gz

Images (two columns: config digest short12 — what docker images shows as IMAGE ID — and the OCI manifest digest):

image	config digest (short)	OCI manifest digest
texera-access-control-service	05f519e63c6d	sha256:d4cf48eff8fc5e0ef6acb0466f7772d775d18662554f018d478cbd538edcafe2
texera-agent-service	cdceeaadc6a0	sha256:594d2ed6c185f052954512e065dc4eb6782e547fae980424d7fe47034d715447
texera-config-service	ce07f59b66bf	sha256:bd01aa0a7be88814022d11071c680591301b545c70252e2ad8651f5c50ee5b97
texera-dashboard-service	8e5bb792bd55	sha256:9833b3a6b21772adf6fcd2fe65491e46cd4ee2ffa014ea36703858d1fe734a5f
texera-file-service	aafd59d22f8b	sha256:a0b089f5a68bfeb01c9dd7075eb93f3a47fd3301ad78798fab459c3fb8365b92
texera-jupyter	6de48c14e4fb	sha256:31c86b04d1359798afa333656485a74981533cd99f7c9b5c4b3383d47664a9f2
texera-mounter	45e96b51168b	sha256:4921af0332562cdc6a0e055fe7128dcad739f6eb584a7ef5ce118fe979ca4d3a
texera-notebook-migration-service	8ea30e49d44f	sha256:f6754786bcccf79248b53cc92b930ae48cee9810fc41e91b6ed748475d0ed59b
texera-workflow-compiling-service	c95f102428b0	sha256:82e39fc586957a6cf577d4eae2403cf8e2cf03053d063ebcb1fbb45b230947c9
texera-workflow-computing-unit-managing-service	aa5de77eeff6	sha256:0eea7702ba7fa6961cc105417acdc10a25b9ccb587881924d50471aae6ccfcb2
texera-workflow-execution-coordinator	3cc022d6097b	sha256:17301301784e0c3bbb9c716bca1c38733763b9630538288ea98f0388b4b7e09d
texera-workflow-execution-runner	825520e0ba7a	sha256:dcac676ffc0d4824c709806f4a1c6a207dfd17a90be854fc4843d86984a3be77
y-websocket-server	fd215376b08c	sha256:ef114c30309c8b0a97c40ae62fd65a4b03b7879c0817294590708817cd10713a

None of these match your values in either column, so I suspect we built or hashed differently rather than hit real non-determinism. Could you share the exact command you ran and which kind of hash your values are? Then we can line the two sides up.

[parshimers]

I used skaffold to build the images. I will play with it some more here. How did you build your containers, just file by file after doing sbt protocGen and compile?

[mengw15]

I followed bin/README.md exactly — sbt DAO/jooqGenerate first (the only host-side step), then one docker buildx build per dockerfile with the flags in my previous comment; the Dockerfiles compile everything inside the build themselves. No skaffold involved.

That probably explains the mismatch: skaffold won't pass the README's pinning flags, so its builds pick up the wall clock, and its hashes are the local docker images IDs. If you rerun with the README invocation, we should line up.

[parshimers]

I tried building the access-control-service image manually and at least for me, it doesn't seem to be consistent with itself. I wonder if Claude fooled itself by not purging the cache between runs, or something. I will investigate more.

[parshimers]

OK, I was able to reproduce everything besides texera-workflow-execution-coordinator. I'm not sure what's going on there, I will have to dig more. The y-websocket-server also seems to not match but that one seems to build differently, so I will have to dig there too.
I verified everything by hand. The issue before was twofold:

• I'm using podman/buildah, not docker/buildx. It has to use buildx, and buildx at the same version.
• If you run from an unclean worktree, the dockerfiles seem to vaccum up other things that aren't in git. That of course changes the end results.

Overall a good result, but a bit more investigation is needed.