Calibrate stop-sequence and token-limit evidence before authority

[halfaipg]

Why

The production preview is exercising the randomized, assignment-bound workload lanes, but two protocol-conformance lanes have materially different behavior from the rest. In a read-only 24-hour production sample on 2026-08-31:

• stop.sequence: 10 healthy / 105 completed; 65 failed results had empty visible output. All completed results reported finish_reason=stop.
• token.limit: 3 healthy / 85 completed; 80 failures reported finish_reason=length, 2 reported finish_reason=stop.
• Exact output, JSON, context retrieval, code, logic, and tool-chain lanes produced substantial healthy evidence in the same window.

This is evidence-only today, so it is not an incident. It is a calibration gate: do not let these two lanes influence routing, rewards, strikes, or model-quality claims until the failure source is separated.

Work

• Add bounded reason codes for scorer failures without persisting raw challenge answers.
• Verify each current text-worker version forwards stop and max_tokens unchanged and preserves visible text, reasoning text, usage, and finish_reason through terminal collection.
• Distinguish backend unsupported/unknown from observed nonconformance; absence of capability must not become a negative quality verdict.
• Run supervised staging probes per backend and compare Grid-observed terminal metadata with the backend-native response.
• Review the gross token-window tolerance against reasoning-model behavior without claiming tokenizer parity.
• Keep both lanes classified as protocol conformance, not general intelligence or model fidelity.

Exit criteria

• Failure reason distribution is observable without answer leakage.
• At least one supported backend passes each lane repeatedly end to end.
• Unsupported backends are not falsely scored as bad models.
• Tests cover empty stop output, forwarded stop text, reasoning-token accounting, and terminal reason preservation.
• Validator economics and routing authority remain disabled.

[halfaipg]

Calibration progress (2026-08-31):

• Core PR validators: classify text probe score failures grid-core#80 is merged and dark-deployed as 97efb358041ee00351e62e65b9b90f24fcf0d7e8. Completed text probe envelopes now carry one private, bounded score_reason; the public verdict/score contract is unchanged and no answer material is included. Full Core CI, PostgreSQL restore proof, backup/cutover checks, and post-deploy health passed.
• Worker PR test(worker): preserve validator protocol fields grid-text-worker#27 is merged. It proves the current structured worker path preserves Grid-issued stop and max_tokens exactly while forcing streaming and requesting usage. All Python versions, Docker, four platform builds, release payload verification, dependency audit, and secret scans passed. This was test-only; no new worker release was published.
• Read-only production grouping confirms the backend split: empty visible stop output is concentrated on reasoning-model backends, while Smollm and qwen38 variants return visible stop output. Token-limit empty output is similarly concentrated on reasoning paths. This supports treating capability/transport uncertainty separately from a general model-quality claim.

Still open: wait for post-deploy assignments to accumulate the new reason distribution, then run supervised backend-native comparisons and decide which backends explicitly support each protocol-conformance lane. Both lanes remain economically inert and excluded from routing/reward claims.

[halfaipg]

Live post-deploy evidence from Core 97efb358 is now available.

Since the 2026-08-31 15:51 UTC deployment, completed text probes are persisting bounded, allowlisted score_reason values with no missing reason field:

• token.limit on gpt-oss-120b: 5/5 failed / empty_visible_output with terminal reason length
• tool.chain on deepseek-v4-flash-nvfp4: 5/5 failed / malformed_output with terminal reason stop
• context.retrieve.32k on qwen38-flash-next-125b-nvfp4: 1/1 healthy / accepted

This confirms the telemetry is working in production and sharpens the calibration problem: failures remain backend/canary-specific rather than a generic validator transport failure. No economic effect is attached. Next useful comparison is the same canary kinds across non-reasoning/native backends before changing scoring policy.

[halfaipg]

Calibration update (2026-08-31): we now have enough production evidence to make the first policy decision without weakening the probes.

• stop.sequence is backend-sensitive but not random noise. In the read-only seven-day aggregate, the gpt-oss path completed 128 probes with no visible committed prefix; the DeepSeek path showed the same empty-visible shape across 51. Two observed Qwen38 aliases completed 40 stop probes, with 26 exact-prefix passes and 14 doubled-prefix failures.
• token.limit is also backend-sensitive. The gpt-oss path commonly consumed hidden reasoning while returning no visible repeated marker. DeepSeek produced a mixed population (15 healthy, 76 failed), so blanket-passing or blanket-skipping the lane would both hide useful API behavior.
• The first 55 envelopes carrying bounded score_reason values produced 35 accepted results and 20 classified failures (malformed_output or empty_visible_output in the observed stop/token/tool-chain lanes). No reason contains answer material.

Decision: keep strict API-level verdicts, but keep these rows classified as protocol_conformance, quality_eligible: false, and economically/routing inert. They must not be presented as proof that a model is unintelligent or lacks fidelity.

A privacy-safe aggregate report is merged in Core PR AIPowerGrid/grid-core#83 (8f87bdf9), and the redacted evidence/policy is recorded in validator PR #72 (b0fd684). The remaining calibration item is bounded backend-native versus Grid-path comparison; no validator binary, release, reward, routing, strike, bond, or slashing change is justified yet.

[halfaipg]

September 13 corrective-release checkpoint:

• Core PR183 is deployed as d606e4d8. The versioned text.token_limit.v2 rule handles a proper partial terminal marker after sufficient complete repetitions, without changing historical v1 semantics or masking malformed output. Read-only replay of 75 retained token-limit responses corrected 35 fragment-related results across nine groups; the other failure classes stayed failures. This is counterfactual replay, not newly completed inference or rewritten history.
• Validator PR115 fixed a separate evidence-delivery bug: a completed, commitment-verified empty response was skipped before signing unless an optional error flag was present. The corrected guard still rejects missing/null output, unavailable probes and invalid bindings. Offline real-capture replay signed/scored 114 completed-empty captures and left 47 unavailable captures unsigned, using mocked delivery only; no expired assignments were resubmitted.
• The combined code is released as immutable v0.1.0-preview.20 (c73a284). Four-platform builds and clean installs, container qualification, exact downloaded payload verification and all eight source-bound provenance checks passed. Windows is unsigned and macOS unnotarized; it remains a preview.
• Core admitted .20 while retaining older reviewed versions. All three first-party services upgraded with their identities/configuration/journals preserved. Fresh tool-chain and 32K-context reports verified independently with zero credit, reservation or worker-reward rows.

Still open: fresh live v2 and committed-empty delivery canaries, repeated backend-native/Grid comparisons, and measured model-fidelity strength. Those ordinary fresh reports do not substitute for the missing lane-specific tests. No compensation campaign, routing authority, penalties or model-identity guarantee was enabled. Keeping this issue open.

[halfaipg]

Fresh preview.20 v2 canary update (2026-09-13 04:48 UTC): the bounded first-party selector obtained a sealed text.token_limit.v2 assignment through the normal assignment API at the existing cooldown boundary, restored the full capability list, and left execution/scoring/signing to the released binary. Two fresh reports (127710, 127711) arrived from owned nodes. Both remain failed: gpt-oss-120b returned reasoning-only output and finish=length, not the required visible repeated markers. Independent retained-body checks verified disclosure seals, prompt/response/evidence commitments, signatures, assignment bindings and reproduced the failed verdicts. No linked worker reward, reservation or credit rows were created. This proves live v2 delivery, not model substitution detection or a healthy token-prefix case. It does NOT exercise fully empty completion delivery because reasoning was present. That separate fresh canary remains open. Documentation PR116 is merged; its snapshot predates this result. No compensation, media, shadow or penalty authority was activated.

[halfaipg]

The remaining fresh empty-completion canary is now verified. At 04:56 UTC September 13, normal sealed stop-sequence assignments produced genuinely empty qwen3-27b replies: no visible text, reasoning, tool calls or worker-error flag. Released preview.20 nodes delivered failed reports 127713/127714 instead of dropping them. Independent retained-body signature, assignment/disclosure, prompt/response/evidence commitment and local-score checks passed; no economic rows were created. The v2 and empty-response delivery gates are both satisfied. Evidence is merged in PR117 / c7e4284. Website PR79 is promoting the combined release after browser CI. This does not identify the cause of the empty backend replies or establish model identity; those limitations and the unpaid, non-punitive boundary remain.

[halfaipg]

Follow-up and correction to the empty-output interpretation: PR #118 records six direct owned-Qwen calls using the two retained stop assignments. All returned HTTP 200. Original-stop streaming and non-streaming produced reasoning but zero visible output. Removing stop produced final answers whose normalized prefixes matched both expected commitments; the marker appeared inside reasoning as well as the final answer.

These controls strongly support stop matching during reasoning, rather than token exhaustion or a streaming-only failure. Budgets were 1024, while original stored usage was only 51/58 completion tokens. Internal decoder traces were not captured, so the exact decoder path remains an inference.

Importantly, deployed Core only retains reasoning_text for token-limit probes. Reports 127713/127714 prove completed empty-visible evidence delivery, NOT that the backend generated zero reasoning. The .20 delivery fix remains valid; earlier fully-empty-backend wording is corrected in the PR. No historical scores, services or economic state changed. The revised stop policy still needs explicit controls and versioned evidence before authority. Raw prompts and responses remain private.

[halfaipg]

PR #118 is now merged as da2d77e after all source/security checks and all four native builds/clean installs passed. Four additional controlled requests used the documented non-reasoning flag on the owned Qwen endpoint: both stop-present answers exactly matched their expected hashes, while both no-stop controls continued through the marker and suffix; all four had zero reasoning. This is a verified control for those two cases on that backend, not a fleet-wide flag rollout. The deployed worker request-builder also preserves the option in an offline component check with no backend calls or service changes.

Core audit follow-up is PR AIPowerGrid/grid-core#188. It separately records direct reproduction of malformed DeepSeek tool arguments; that is a backend-output defect and must not be fixed by permissive JSON scoring. No runtime, historical scores or economic controls changed. Stop-policy deployment, operator review/consent and compensation approval remain separate.