Problem
Bundle CI runs the full upstream test suites (valkey core, json, bloom) against
every image after it is built. It is slow and flaky, and it delays releases.
Two reasons it is breaks more than normal flakiness:
The suites run in a mode they were not designed for. Upstream spawns
a server per test class so each class sets its own config; the bundle points every
class at one shared container.
Classes that assume a server config break here even though they pass upstream
which is why run.sh keeps growing --skiptest and grep -v lists.
We re-test code that already passed at the exact tag we consume, using suites that do not target
what is actually bundle-specific (our toolchain, module co-loading, the container runtime).
Proposal
Test the bundle's module combination at module release time: when a module releases,
run its suite against a server with all bundle modules loaded, in that module's own CI
where the suite runs natively. Co-loading regressions then surface before the version
reaches versions.json.
Bundle CI keeps the build (this is what catches real integration breaks) plus a fast smoke layer: modules load with expected versions, one real command each, RDB save/reload with module data, and an upgrade smoke starting the previous image's data dir on the new image (the drop-in-replacement contract, untested today). Full upstream suites move to a nightly job.
Outcome
Release latency drops from hours to minutes, failures point at real bundle problems, and skip lists stop growing with every upstream test addition.
Problem
Bundle CI runs the full upstream test suites (valkey core, json, bloom) against
every image after it is built. It is slow and flaky, and it delays releases.
Two reasons it is breaks more than normal flakiness:
The suites run in a mode they were not designed for. Upstream spawns
a server per test class so each class sets its own config; the bundle points every
class at one shared container.
Classes that assume a server config break here even though they pass upstream
which is why run.sh keeps growing
--skiptestandgrep -v lists.We re-test code that already passed at the exact tag we consume, using suites that do not target
what is actually bundle-specific (our toolchain, module co-loading, the container runtime).
Proposal
Test the bundle's module combination at module release time: when a module releases,
run its suite against a server with all bundle modules loaded, in that module's own CI
where the suite runs natively. Co-loading regressions then surface before the version
reaches versions.json.
Bundle CI keeps the build (this is what catches real integration breaks) plus a fast smoke layer: modules load with expected versions, one real command each, RDB save/reload with module data, and an upgrade smoke starting the previous image's data dir on the new image (the drop-in-replacement contract, untested today). Full upstream suites move to a nightly job.
Outcome
Release latency drops from hours to minutes, failures point at real bundle problems, and skip lists stop growing with every upstream test addition.