perf(stratum-apps): make TaskManager::spawn tracking constant-time

[gimballock]

Relates to #492; not Closes.

This removes one candidate cause of the plateau (analysis). The attribution is inferred, not confirmed: the measurement below was taken on a patched tree, and nothing here rules out other contributors. #492 should stay open until the plateau is confirmed gone on stock upstream.

Problem

TaskManager::spawn pruned its Vec<JoinHandle<()>> with retain(|h| !h.is_finished()) on every spawn — a linear scan of every live handle, taken under the tracking mutex. Callers spawn several tasks per downstream connection, so reaching N concurrent connections costs O(N²) and serializes that scan on one lock.

The observable symptom is a connection-accept plateau at low total CPU, because a single core saturates on the scan while the rest idle.

Fix

Track tasks in a tokio::task::JoinSet. Insertion is constant time. Because a JoinSet releases an entry only when that entry is joined, and join_all runs only at shutdown, spawn also drains already-finished tasks with the non-blocking try_join_next. That drain touches only tasks the runtime has already marked complete, costs O(tasks finished since the previous spawn) — amortized constant per task — and keeps the set bounded to roughly the live task count.

The drain is load-bearing rather than an optimisation: without it the set would retain one task control-block per task ever spawned, growing with cumulative rather than concurrent tasks.

Measurement

r7i.4xlarge, pool isolated on its own host, one variable changed:

	connections	pool CPU
Vec + retain	plateau at 34,543	25.3%
JoinSet	61,259	99.6%

The measured tree was this repo vendored at 14bb42d888fb (an ancestor of main) with a local patch stack, so it was not stock upstream — but the tracking change was the only variable between the two runs. The measurement covers the first commit only; the second postdates it.

Behavioural changes to be aware of

Nothing in this repo relies on any of these, but they are real deltas:

• join_all awaits in completion order, not reverse-spawn order. The previous doc comment promised the latter; join_next does not provide it, and no caller depends on ordering.
• abort_all leaves aborted entries in the set until they are joined, where the old drain(..) emptied it. Callers wanting them reaped should follow with join_all, which shutdown already does.
• The mutex is retained. This removes the per-spawn scan, not the lock; spawns still serialize on it. It simply stops being the binding constraint at this scale.

Commits

1. perf(stratum-apps): make TaskManager::spawn tracking constant-time — the change the measurement above covers, plus three tests (TaskManager had none): join_all drains every entry, spawn releases finished entries so the set stays bounded under churn, and a panicking task is still reaped.
2. feat(stratum-apps): surface tasks that exit abnormally — both the old Vec and the new JoinSet discard join results, so a panicking task vanished silently. This logs a warning for non-cancellation join errors, collecting them under the lock and reporting after release, because reporting inline would put subscriber work on the contended path and would deadlock if a subscriber ever spawned through this manager. Split out deliberately so it can be dropped without touching the measured fix.

Notes

• The bounded-set test runs on a current-thread runtime with batches kept under the scheduler's per-tick poll budget. A multi-threaded variant passed in isolation but failed under the parallel suite, which is worth knowing if this test is ever revisited.
• No version bump: stratum-apps is at 0.8.0 while crates.io has 0.7.0, so the crate is already bumped since last publish, and this change is API-compatible (per CONTRIBUTING factor 2).
• No new dependency; tokio already carries full, which includes JoinSet.

Validation

Run locally against main (2e40be08) on the pinned 1.85.0 toolchain:

• cargo +nightly fmt --all --manifest-path=stratum-apps/Cargo.toml -- --check — pass (CI formats with nightly)
• cargo clippy --manifest-path=stratum-apps/Cargo.toml --all-features -- -D warnings — pass
• cargo test --manifest-path=stratum-apps/Cargo.toml --all-features — 142 passed, stable across repeated runs

I did not run the other manifests' clippy/fmt or the integration tests locally; the change is confined to one file in one crate with no API or dependency change.

[plebhash]

Closes #492 (candidate cause — see #492 (comment)).

if the root cause is speculative, we shouldn't close the issue because that would imply certainty that the issue has been permanently solved

[gimballock]

Agreed — dropped the Closes. The attribution is inferred (patched tree, no stock-upstream reproduction), so #492 stays open.