Skip to content

SPEC-028 PR-B: greedy gate + /v1/status telemetry + AC harnesses#398

Merged
Augustas11 merged 2 commits into
mainfrom
codex/spec-028-pr-b
Jul 5, 2026
Merged

SPEC-028 PR-B: greedy gate + /v1/status telemetry + AC harnesses#398
Augustas11 merged 2 commits into
mainfrom
codex/spec-028-pr-b

Conversation

@Augustas11

Copy link
Copy Markdown
Owner

Summary

Implements SPEC-028 PR-B "Telemetry + greedy gate" on top of PR-A plumbing (03522c4):

  • FR-5 greedy gate. Speculative generation runs only when request.temperature == 0.0 AND the request has no feature the Swift speculative primitive cannot verify (see ChatCompletionRequest gating). Stochastic and featureful requests stay on the existing TokenIterator path — buyer output is unchanged.
  • FR-8 /v1/status telemetry. Adds spec_decode_enabled, spec_decode_draft_model_id, spec_decode_num_draft_tokens, spec_decode_drafted_tokens_since_last, spec_decode_accepted_tokens_since_last, spec_decode_acceptance_rate. Aggregated over the same window as request/tps; window resets on each metrics roll.
  • FR-9.5 warm-swap counter coherence. spec_decode_drafted/accepted_since_last reset to zero on target-swap boundary so acceptance rate never blends pre/post-swap draft populations.
  • Opt-in-only heartbeat. Coordinator heartbeat carries the FR-8 fields only when the operator opts in; coordinator forward-compat tests assert the unknown-field-tolerant path.
  • AC-10 executable canary. macprovider-cli spec028-canary ac10 harness with fail-closed 16 GB Apple Silicon guard and pinned Qwen2.5-Coder-7B+1.5B fixture (spec028-code-iso8601-v1.json). AC-11 M1 8 GB harness (Spec028PlumbingTests/testSmallAirLlama32CanaryWhenExplicitlyEnabled) gated by SPEC028_RUN_SMALL_AIR_CANARY=1 and host-memory ≤ 12 GB.
  • AC-9a speculative failure tests. Warm-swap and speculative-failure paths covered end-to-end without touching the SPEC-015 v0.4 receipt tuple (FR-7 preserved).

Validation

  • git diff --check
  • swift test (all): 890 tests, 8 skipped, 0 failures.
  • Focused Swift filter set: 135 tests, 1 expected AC-11 skip, 0 failures.
  • go test ./internal/ws (heartbeat forward-compat): pass.

BUILD audit

Three-lane Codex BUILD audit loop completed:

  • CODE C/H/M: 0/0/0
  • SECURITY C/H/M: 0/0/0
  • ARCH C/H/M: 0/0/0

Known validation gaps

Both live-hardware acceptance criteria are pinned in code and env-gated, but were not executed on the required hosts. Each becomes a hard merge gate on PR-C (warm-swap), at which point spec decode first becomes operator-visible in the intended enable-in-prod path. FR-1 disabled-mode byte-identity keeps the beta fleet unaffected while PR-B lands: no beta/config-*.yaml sets draft_model, and the flag is undocumented in operator materials until PR-C.

AC-10 — 16 GB M4 performance canary (waived → PR-C)

  • Requirement: ratio-over-baseline (≥ 1.4× median speedup, ≥ 0.30 acceptance) plus sustained thermal window (≥ 1.2× baseline over last minute of a 5-minute run) on real M4 16 GB Apple Silicon with Qwen2.5-Coder-7B+1.5B pair.
  • Status: no 16 GB M4 Air available at PR-B open time. The spec028-canary ac10 harness fail-closes on non-16 GB hosts by design; diagnostic-mode runs on other hardware do not satisfy AC-10 (harness documents this).
  • Gate on PR-C: PR-C MUST include an ## AC-10 evidence block with a real 16 GB M4 run, or PR-C is blocked from merge.

AC-11 — M1 8 GB Air load canary (waived → PR-C)

  • Requirement: target mlx-community/Llama-3.2-3B-Instruct-4bit + draft mlx-community/Llama-3.2-1B-Instruct-4bit complete short_chat and streaming_check at temperature=0 on real M1 8 GB Air without Metal OOM or process crash. Validates FR-4.6 ProviderCapacity headroom refresh.
  • Status: attempted on an M1 8 GB Air running macOS 26.3.1; blocked by mlx-swift 3.31.4 SwiftPM .xctest-bundle metallib-resource resolution failure at test-runtime (metallib compiles but XCTest bundle cannot locate it via Bundle.module; not a repo issue). Diagnostic reproduces every run: MLX error: Failed to load the default metallib at mlx-swift/Source/Cmlx/mlx/c/mlx/c/stream.cpp:115. Requires macOS ≤ 15 host or a workaround for the SwiftPM test-bundle resource lookup on macOS 26.
  • Gate on PR-C: PR-C MUST include an ## AC-11 evidence block from real M1 8 GB Air hardware, or PR-C is blocked from merge. Options being coordinated for PR-C: beta operator loan, fleet 8 GB M1 host reflashed to macOS 15, or a PR-C-scoped macprovider-cli spec028-canary ac11 executable that sidesteps the .xctest bundle-resource bug (executable targets don't hit the same lookup path).

Blast radius while gates are open

  • No beta/config-*.yaml sets draft_model. Existing providers are byte-identical (FR-1).
  • The only way to enable spec decode after PR-B is an operator manually setting --draft-model via CLI/env/YAML. That path fail-closes at startup on capacity shortfall (FR-4.6 draft_model_capacity_shortfall) and on tokenizer mismatch (FR-4.4 draft_model_tokenizer_mismatch) — a misconfigured provider process refuses to start rather than serving corrupt traffic.
  • Buyer API surface and SPEC-015 v0.4 receipt tuple are unchanged (FR-7). No settlement verifier impact.
  • Coordinator routing preferences are unchanged. Heartbeat forward-compat tests assert unknown-field tolerance so the opt-in heartbeat fields do not force a coordinator-side schema update.

Closes SPEC-028 §8 PR-B scope. PR-C (warm-swap lifecycle, FR-9.1–9.4, AC-9) is next.

…ness

Adds FR-5 temperature==0 greedy gate (stochastic requests stay on TokenIterator),
FR-8 /v1/status spec_decode_* fields with window reset on target swap, opt-in-only
heartbeat forwarding with coordinator forward-compat, the executable
macprovider-cli spec028-canary ac10 harness with fail-closed 16 GB Apple Silicon
guard, and an AC-11 M1 8 GB harness gated by SPEC028_RUN_SMALL_AIR_CANARY=1.

Constraint: SPEC-028 PR-B requires FR-5, FR-8, AC-5, AC-7, AC-8, AC-10, and AC-11 coverage before PR opening.
Rejected: opening a PR from this session | AC-11 real M1 8 GB Air evidence is unavailable on the current host.
Confidence: high
Scope-risk: moderate
Directive: Do not open PR until AC-11 M1 8 GB Air evidence is recorded in the PR body; AC-10 real M4 16 GB canary evidence is also still needed.
Tested: swift test (890 tests, 8 skipped, 0 failures); focused Swift SPEC-028 test lane (135 tests, 1 skipped, 0 failures); go test ./internal/ws; git diff --check; macprovider-cli spec028-canary host guard failed closed on 32 GB host; three-lane BUILD audit 0/0/0.
Not-tested: AC-10 real M4 16 GB Qwen2.5-Coder-7B+1.5B canary; AC-11 real M1 8 GB Air canary.
@Augustas11 Augustas11 marked this pull request as ready for review July 5, 2026 12:56

@antfleet-ops antfleet-ops left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR-B telemetry + greedy gate: three-lane BUILD audit 0/0/0, swift test 890/0 (8 skipped), go test ./internal/ws pass. Scope matches SPEC-028 §8 PR-B (FR-5, FR-8, FR-9.5). Dual-waiver on AC-10/AC-11 documented in body with explicit PR-C merge-gate; FR-1 disabled-mode byte-identity + FR-4.6 fail-closed capacity shortfall bound blast radius while gates are open. Coordinator heartbeat forward-compat tests present. LGTM.

@Augustas11 Augustas11 merged commit ccc16be into main Jul 5, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants