Repository navigation
Conversation
## Summary - The benchmark's `scale` / `down` row was not a like-for-like comparison (#1947). After `up -d --scale app=5`, podman-compose 1.6.0's `down -v` exits 0 in about 0.38 s but only stops and removes `app_1`: `app_2` to `app_5`, the pod `pod_<proj>` and the network `<proj>_default` stay behind. podup removes all five and the network in about 0.26 s on the same host. The 5.10.5 table therefore showed podup losing a row it actually wins, and the leftovers stayed on the host and slowed the rows measured after them. - After every timed `down`, the harness now checks the engine for containers, pods and networks of the project. A tool that exited 0 but left something behind gets that row recorded with rc 97, which the aggregator already excludes from the statistics and reports as `[F failed of N]`; the leftovers are force-removed so the next row starts clean. ## Changes - `bench/leftovers.sh` (new, sourced by `run.sh`): detection by each tool's compose project label, verified on real resources (`podup.project`, `io.podman.compose.project`, `com.docker.compose.project`), plus an exact-name match for podman-compose's unlabelled `pod_<proj>`. Engine queries go to the same Podman socket and Docker host the measured tools use. A failing engine query is reported, never read as "clean"; a purge is re-checked and anything still there is printed. - `bench/run.sh`: `record_down` wraps the timed `down` of the `updown` and `scale` ops; a non-zero tool rc is kept, only a zero rc with leftovers becomes 97. `teardown` checks and cleans between scenarios and says so on stderr. - `bench/aggregate.py`: a legend line under the tables and a `_notes` entry in `summary.json` whenever rc 97 rows exist; the self-test fixtures include two such rows and assert the legend. - `tests/shell/bench-leftovers.test.sh` (new, wired into `lint-shell.yml`): stub `podman`/`docker` that reject the invalid `{{.Names}}` field on `pod ls`/`network ls` like the real CLIs and log every removal call. ## Test plan - [x] `bash tests/shell/bench-leftovers.test.sh`: 28 passed - [x] `bash tests/shell/ci-runs-every-test.test.sh`, `check-planning-words`, and shellcheck at `--severity=style` over every tracked shell file: clean - [x] `python3 bench/aggregate.py --self-test`: ok - [x] Sabotage: putting `{{.Names}}` back, dropping the label filter, and rewriting every leftover rc to 97 each made a named case fail - [x] Full run on Podman 5.7.0 (`bench/run.sh --iters 1 --warmup 0 --engines podman --allow-dynamic`): only podman-compose `scale` `down` was flagged (rc 97, with the pod, the four containers and the network in the warning); podup and docker-compose-podman were clean in every scenario, including `network-ipam`'s custom network; no `bench_` resources left afterwards - [x] By hand: a podman-compose `--scale 5` project torn down with `down -v`, then `bench_check_leftovers` reported `pod_… app_2 … app_5 …_default`, returned 97 and left nothing behind - [ ] Podman lane (not affected: no Rust change) ## Checklist - [x] Targets `develop` - [x] Commits are signed off and signed - [x] Labels applied - [x] Every new file in `tests/shell/` is named in a workflow's `test-command` - [x] No secrets, keys or credentials ## Related issues Fixes #1947 once the release pull request merges into `main`. The published benchmark numbers need a fresh run with this harness; I will do that separately. --------- Signed-off-by: Jaro-c <75870284+Jaro-c@users.noreply.github.com>
Bumps `Cargo.toml`, `Cargo.lock` and `debian/changelog` to 5.10.8 together. Patch release carrying #1965 (#1947): the benchmark harness fails a `down` row that left the project's resources behind. No change to the podup binary's behaviour. Signed-off-by: Jaro-c <75870284+Jaro-c@users.noreply.github.com>
Jaro-c
added
prio:P2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 5.10.8: no change to how podup behaves. The benchmark harness now fails a
downrow when the tool exited 0 but left containers, pods or networks of its project behind, and removes them before the next row (#1947, #1965). The publishedscale downcomparison had podman-compose 1.6.0 leaving four of five replicas running; the benchmark will be re-run on this release.Closes #1947