Skip to content

docs(validator): separate backend failures from probe calibration - #188

Merged
halfaipg merged 3 commits into
mainfrom
docs/validator-backend-failure-diagnosis
Sep 13, 2026
Merged

docs(validator): separate backend failures from probe calibration#188
halfaipg merged 3 commits into
mainfrom
docs/validator-backend-failure-diagnosis

Conversation

@halfaipg

@halfaipg halfaipg commented Sep 13, 2026

Copy link
Copy Markdown

What

Update the original failed-group audit with deployed token-limit v2, controlled stop/reasoning tests, direct DeepSeek and GPT-OSS tool-format comparisons, and an independent oracle for the four logic failures.

Evidence

  • Read-only bounded PostgreSQL queries retain the original twelve larger-model tool failures and four larger-model logic failures.
  • Six direct DeepSeek controls reproduce three malformed argument strings byte-for-byte; auto mode leaks native markup despite JSON arguments.
  • Six actual GPT-OSS controls return five missing calls and one correct first-stage call. Three failures contain the correct argument object as text, not a structured call. A separate failed helper attempt with an empty auth header is retained and excluded.
  • Four non-reasoning Qwen controls validate stop behavior on the owned backend, not fleet-wide compatibility.
  • Independent arithmetic parsing verifies all four logic commitments and confirms wrong-integer answers.

Verification

Independent retained-response parsers and expected-commitment comparisons; offline deployed-worker request-builder check; diff-check and staged secret scans passed. Updated-head full CI remains required. Private prompts, responses and credentials stay outside git.

Risk and next steps

Docs only: no services, scores, historical records or money state changed. No permissive JSON scoring or blanket provider options. Endpoint-reported versions are not verified runtime artifacts. Actual backend host access, pinned backend qualification and full tool-chain tests remain necessary. Compensation budget, independent-operator reviews and human consent are still pending.

@halfaipg
halfaipg merged commit 6c57c3a into main Sep 13, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant