Skip to content

Commit ec24b85

Browse files
committed
feat: add label-gated skill evaluation with cumulative PR budget
Keep ledger authority in the trusted controller and gate every provider request. Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com> Agent-assisted-by: OpenAI Codex
1 parent d9dc091 commit ec24b85

17 files changed

Lines changed: 2139 additions & 134 deletions

‎.github/workflows/skill-eval-pr.yml‎

Lines changed: 10 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,11 @@ run-name: >-
77
on:
88
pull_request_target:
99
branches: [main]
10-
types: [opened, synchronize, reopened]
10+
types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled, closed]
1111
paths:
1212
- src/inferrsi/data/skill_library/**
13+
- src/inferrsi/data/knowledge_packs/**
14+
- src/inferrsi/data/repo_packs/**/knowledge/**
1315
- tools/skill_eval_*
1416
- docs/examples/skill-comparison/**
1517
- .github/workflows/skill-eval-pr.yml
@@ -58,7 +60,8 @@ jobs:
5860
plan:
5961
outputs:
6062
client_kind: ${{ steps.prepare.outputs.client_kind }}
61-
if: github.ref == 'refs/heads/main'
63+
auto_candidate: ${{ steps.prepare.outputs.auto_candidate }}
64+
if: github.ref == 'refs/heads/main' && github.event.action != 'closed'
6265
runs-on: ubuntu-latest
6366
timeout-minutes: 10
6467
permissions:
@@ -79,11 +82,6 @@ jobs:
7982
env:
8083
GITHUB_TOKEN: ${{ github.token }}
8184
run: python -m tools.skill_eval_pr prepare --output prepared
82-
- name: Validate explicit operator authorization and reserve execution
83-
if: github.event_name == 'workflow_dispatch' && inputs.action == 'execute'
84-
env:
85-
GITHUB_TOKEN: ${{ github.token }}
86-
run: python -m tools.skill_eval_pr authorize --output prepared
8785
- uses: actions/upload-artifact@v4
8886
with:
8987
name: skill-eval-plan
@@ -92,12 +90,12 @@ jobs:
9290
retention-days: 30
9391

9492
execute:
95-
if: github.event_name == 'workflow_dispatch' && inputs.action == 'execute'
93+
if: github.event_name == 'pull_request_target' && needs.plan.outputs.auto_candidate == 'true'
9694
needs: plan
9795
runs-on: ubuntu-latest
9896
timeout-minutes: 20
9997
permissions:
100-
contents: read
98+
contents: write
10199
actions: read
102100
pull-requests: read
103101
steps:
@@ -113,21 +111,11 @@ jobs:
113111
with:
114112
name: skill-eval-plan
115113
path: prepared
116-
- name: Existing Codex client preflight
117-
if: needs.plan.outputs.client_kind == 'codex'
118-
run: codex --version
119-
- name: Recheck live revision and authorization before execution
114+
- name: Trusted cumulative-budget controller (never executes PR code)
120115
env:
121116
GITHUB_TOKEN: ${{ github.token }}
122-
run: python -m tools.skill_eval_pr authorize --output prepared
123-
- name: Execute existing Codex comparison without publication credentials
124-
if: needs.plan.outputs.client_kind == 'codex'
125-
run: python -m tools.skill_eval_pr execute --output prepared
126-
- name: Execute configured Yibuapi comparison without publication credentials
127-
if: needs.plan.outputs.client_kind == 'anthropic-messages-agent-v1'
128-
env:
129117
SKILL_EVAL_YIBUAPI_KEY: ${{ secrets.SKILL_EVAL_YIBUAPI_KEY }}
130-
run: python -m tools.skill_eval_pr execute --output prepared
118+
run: python -m tools.skill_eval_pr control --output prepared
131119
- name: Package retained results even after interruption
132120
if: always()
133121
run: python -m tools.skill_eval_pr pack --output prepared
@@ -163,6 +151,7 @@ jobs:
163151
run: python -m tools.skill_eval_pr prepare --output prepared
164152
- name: Update the single revision-bound summary from retained data
165153
env:
154+
SKILL_EVAL_AUTO_SOURCE: ${{ github.event_name == 'pull_request_target' && needs.plan.outputs.auto_candidate == 'true' }}
166155
GITHUB_TOKEN: ${{ github.token }}
167156
run: python -m tools.skill_eval_pr publish --output prepared
168157
- uses: actions/upload-artifact@v4

‎docs/skill-eval-budget-core.md‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
# Cumulative PR budget and Git ledger
2+
3+
Refs [#215](https://github.com/InferMatrixLab/InferRSI/issues/215).
4+
`tools/skill_eval_budget.py` implements pure accounting transitions.
5+
`tools/skill_eval_budget_git.py` implements the GitHub Git-data adapter using the
6+
existing authenticated `GitHub.request` client and injected `GITHUB_TOKEN`.
7+
The adapter never loads credentials or executes PR code. Tests use mocks only.
8+
9+
## Accounting
10+
11+
- Key: immutable repository ID plus PR number; never branch name, repository
12+
name, head revision, workflow run or label epoch. Renames, pushes, retries and
13+
label removal/reapplication cannot create a fresh PR allowance.
14+
- The requested ceiling is USD 200 per PR, represented as 200,000,000 integer USD
15+
microdollars. Trusted pricing calculations must round worst-case costs upward.
16+
Floats, booleans, negative costs and zero-cost reservations are rejected.
17+
- A trusted frozen-plan digest covers all revisions, mapping, exam, provider,
18+
pricing, retries, request/token limits and other cost-relevant inputs. The core
19+
receives its independently computed worst-case total; it does not derive cost
20+
or prove that the plan producer is trustworthy.
21+
- Reserve the entire plan before execution. Charge = audited legacy obligation
22+
plus **every reservation**, including cancelled, failed, interrupted, completed
23+
and ambiguous executions. Actual cost is evidence only; there are no refunds.
24+
- An exact plan/run/cost binding is reserve-idempotent. Reusing a plan with a
25+
different run, or a run with a different plan/cost, fails closed. This first
26+
version deliberately does not support rerunning an identical plan. A distinct
27+
authorized attempt must have a distinct frozen identity and a fresh reservation.
28+
Run identity includes workflow run attempt, not just workflow run ID.
29+
30+
```text
31+
verified genesis → reserved → claimed → finished
32+
└───→ cancelled
33+
```
34+
35+
Claim is non-idempotent: at most one process can obtain a successful claim CAS.
36+
No takeover, lease expiry, automatic retry or reset exists. A process crash after
37+
claim burns capacity even when no request happened. `claim_once` returns only on
38+
an explicit, durable CAS success for that invocation. Its return is **not** a
39+
replayable execution credential. The trusted caller must execute at most once,
40+
respect all frozen bounds (including provider retries), and not expose a result
41+
that another worker can reuse to execute again. An actual-cost overrun is rejected
42+
and requires external quarantine/reconciliation before further paid activity;
43+
preventing overruns depends on the future bounded executor.
44+
45+
## Genesis is an audited migration, never a missing-file default
46+
47+
`Genesis(evidence, legacy_charge)` requires an explicit audit reference and an
48+
explicit integer charge. Zero is acceptable only when the authorized audit proves
49+
no earlier paid obligations. Include settled legacy spend and conservative upper
50+
bounds for all outstanding/ambiguous legacy runs; quiesce or reconcile older
51+
writers before migration. Unknown obligations block genesis. If the obligations
52+
exceed the ceiling, initialization fails and the PR is ineligible.
53+
54+
Constructing this value is not authorization or cryptographic proof. The separately authorized
55+
trusted initializer must verify the audit and atomically create only for a key
56+
that has **never** had a ledger. There is intentionally no genesis initializer in the core or Git adapter. Missing/corrupt existing state, HTTP 404, expired artifacts, empty
57+
workflow history and label reapplication are never permission to initialize.
58+
59+
## Atomic storage contract
60+
61+
`AtomicStore.read_verified(key)` returns a `Snapshot(history_token, ledger)` only
62+
from authenticated, durable, current history. `decode` validates schema and local
63+
invariants, **not provenance, rollback resistance or history completeness**.
64+
The backend retains an independently configured genesis/checkpoint anchor that
65+
survives ledger ref deletion. A ledger's own revision cannot detect its reset.
66+
See the precise rollback trust boundary below.
67+
Missing, corrupt, untrusted or discontinuous history must raise and block work.
68+
69+
`compare_and_swap(snapshot, successor)` must atomically append exactly one
70+
successor against the current immutable history token, using
71+
`validate_successor` as well as key/lineage checks. All writers use the same CAS
72+
boundary. A matching body already stored by another request is **not** this
73+
caller's successful commit. On conflict, reread verified state and recompute;
74+
never overwrite a newer charge. On unknown write outcome, do not execute.
75+
76+
Reservation writes can reconcile by rereading the exact binding and charge.
77+
Claim writes cannot reconcile into a new execution grant: an observed `claimed`
78+
entry may already have executed. Duplicate/ambiguous claims stay blocked. If a
79+
verified read still shows `reserved`, a new CAS can compete safely only against
80+
that current token; a delayed earlier write must conflict with the same boundary.
81+
Terminal evidence retries are idempotent only when identical; no terminal update
82+
changes charged capacity.
83+
84+
## Git-data adapter and trusted bootstrap
85+
86+
Each PR uses exactly `refs/heads/skill-eval-budget/<repository_id>/<pr_number>`.
87+
A trusted `LedgerAnchor` supplies immutable repository/PR identity, expected
88+
`genesis_commit`, and independently retained `checkpoint_commit`. These values
89+
must come from reviewed root configuration, **never PR files, artifacts, labels,
90+
comments, caller-selected refs or ledger contents**. The API repository ID is
91+
checked on every verified read; repository renaming does not refresh capacity.
92+
93+
Before activation an authorized operator must quiesce previous paid writers,
94+
audit settled **and outstanding/ambiguous** legacy obligations, and create a
95+
genesis commit with exactly `ledger.json` holding revision-zero state and the
96+
explicit `Genesis(evidence, legacy_charge)`. The genesis may be parentless or have
97+
exactly one explicitly audited existing-main parent when the authorized Git API
98+
requires a parent. Its tree must still contain **only** `ledger.json`; do not
99+
inherit the main tree. Pin the complete genesis commit SHA and checkpoint in
100+
trusted root configuration. That immutable SHA binds the genesis parent as well
101+
as its state. Parent ancestry before genesis is provenance, not ledger history:
102+
verification stops at the pinned audited genesis and never interprets main commits
103+
as budget transitions. Merge genesis commits are rejected. All commits after
104+
genesis still require exactly one parent and one valid accounting transition. An empty ledger is valid only after an explicit no-prior-spend
105+
audit; unknown obligations block bootstrap. Bootstrap/ref creation is deliberately
106+
not automated by this patch and no missing-ledger fallback exists. If the PR had
107+
an earlier ledger, recover/reconcile that history rather than creating new zero
108+
capacity. Unknown/reset history requires operator investigation, not retry-reset.
109+
110+
Reads fetch the exact ref, every single-parent commit and exact ledger tree/blob
111+
back to pinned genesis. They check all accounting transitions and require both
112+
the configured checkpoint and the last tip observed by that store instance to
113+
be ancestors. Missing refs/objects, merges, extra tree entries, corruption,
114+
discontinuous transitions, divergent roots, rollback behind the checkpoint,
115+
identity changes, racing reads and excessive history all block execution.
116+
History verification is bounded to 10,000 commits and 4 MB per ledger; reaching
117+
bounds fails closed, never truncates accounting or resets capacity.
118+
119+
Writes create an immutable tree and a single-parent child of the verified tip,
120+
then PATCH the existing ref with `force: false`. The
121+
[GitHub reference API](https://docs.github.com/en/rest/git/refs#update-a-reference)
122+
requires a fast-forward when force is false. Under append-only writers, two
123+
children of the same tip cannot both become the next tip. A random nonce makes
124+
even identical state transitions different commits, preventing two identical
125+
claims from receiving successful idempotent updates for a shared commit SHA.
126+
Stale snapshots return conflict; failed/uncertain writes return unknown, including
127+
permission/rules rejection. No automatic PATCH retry or claim reconciliation
128+
confers execution permission. Unreachable trees/commits after a crash do not
129+
change accounting; a committed reservation/claim remains permanently charged.
130+
No branch creation, force update, deletion or repository-settings API is used.
131+
132+
### Trust boundary: not protection against malicious administrators
133+
134+
Git refs are mutable by sufficiently privileged principals. The security model
135+
requires trusted root configuration/code and repository administrators, plus no
136+
out-of-band force writes/deletion/recreation of the ledger ref. Use independently
137+
administered append-only restrictions where available; this implementation does
138+
not change settings or bypass rules. A static genesis pin detects a replaced
139+
root but **cannot detect rollback to a valid later ancestor across new processes**.
140+
A trusted checkpoint detects rollback before that checkpoint, and process memory
141+
detects rollback behind a tip already seen in that process. Neither is an external
142+
monotonic witness. The tests explicitly demonstrate this limitation.
143+
144+
Absolute rollback resistance against malicious repository administrators would
145+
require separately authorized, independently durable monotonic storage or an
146+
independently administered enforcement boundary. Do not claim this Git branch
147+
provides it. Retain and advance reviewed checkpoints as audit evidence; never
148+
lower or replace a checkpoint to make a failed verification pass. This document
149+
does not authorize any settings changes, migration, or remote writes.
150+
151+
## Verification and integration boundary
152+
153+
`pytest tests/test_skill_eval_budget.py tests/test_skill_eval_budget_git.py`
154+
covers cumulative limits, permanent reservations, legacy charges, duplicate
155+
plan/run bindings, malformed state, reservation/claim races, unique claim commit
156+
identities, crashes/lost responses, rules rejection, history corruption/deletion,
157+
checkpoint rollback detection and its explicit administrator limitation. The
158+
mock Git API enforces non-force fast-forward semantics. CPU tests do not prove
159+
provider billing or live GitHub rules/permission compatibility.
160+
161+
The adapter does not create a plan, grant label authority, run a provider or
162+
reconcile actual billing. The trusted controller must reserve and claim before
163+
executing, revalidate admission, enforce all worst-case plan bounds including
164+
retries, and keep credentials away from untrusted PR code. A returned claim is
165+
process-local permission for one execution, not a reusable workflow output.
166+
Live activation still requires the audited genesis/root configuration and
167+
successful permitted writes; absent prerequisites fail closed.

0 commit comments

Comments
 (0)