Skip to content

Commit a5818e6

Browse files
committed
Fix delivery history preservation and browser CI coverage
Preserve cached final and search assessments with explicit unverified provenance when retained observations are unavailable or conflicting. Keep current acceptance fail-closed and run both offline delivery report cases in browser CI. Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
1 parent e2e9d08 commit a5818e6

6 files changed

Lines changed: 209 additions & 26 deletions

File tree

‎.github/workflows/ci-ready.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,12 +70,13 @@ jobs:
7070
run: |
7171
python -m pip install -e ".[dev]" playwright
7272
python -m playwright install --with-deps chromium
73-
- name: Test dashboard navigation and live refresh in Chromium
73+
- name: Test dashboards and offline delivery reports in Chromium
7474
run: |
7575
python -m pytest -vv -o addopts='' \
7676
tests/test_profile_story_checks.py::test_profile_and_trace_shortcuts_open_recorded_evidence \
7777
tests/test_knowledge_navigation_browser.py \
7878
tests/test_dashboard_experiments_browser.py \
79+
tests/test_result_delivery.py::test_offline_html_remains_readable_without_scripts_or_network \
7980
--junitxml="$RUNNER_TEMP/dashboard-browser-results.xml"
8081
- name: Retain browser test results
8182
if: always()

‎docs/result-delivery.md‎

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -51,9 +51,12 @@ acceptance remain separate.
5151
as an audit fact and sets `delivery_accepted: false`.
5252
3. **Degraded reads:** a separate read-only adapter never weakens Harness `_load`
5353
or reassesses evidence. An intact digest-bound observation retains historical
54-
PASS when measurement bytes are missing or modified. A missing, invalid or
55-
changed observation cannot establish PASS from its unverified cached assessment;
56-
the report says “Not assessed” and references preserved state for audit.
54+
PASS when measurement bytes are missing or modified. If the observation is
55+
missing, invalid or conflicts with committed state, a readable cached assessment
56+
in `state.json` retains the historical verdict and completion facts, explicitly
57+
labelled **cached state; unverified**. It cannot establish current delivery
58+
acceptance. If neither source supplies a readable assessment, the report says
59+
“Not assessed”; it never invents a verdict or substitutes a search PASS for final.
5760
4. **Snapshot:** native exports share the controller's nonblocking OS lock. Since
5861
writers hold it throughout an attempt, progress exports happen between committed
5962
operations. A second digest pass checks every source read, negative lookup and
@@ -100,7 +103,24 @@ A missing/interrupted final stays incomplete. A final snapshot means the source
100103
stopped, not that it succeeded. Historical completion survives later evidence loss;
101104
prominent current gaps prevent current delivery acceptance.
102105

103-
The primary assessment is the final observation, never a favorable search result.
106+
The primary assessment is the final record, never a favorable search result.
107+
For native records, `assessment_provenance` on acceptance, final, checks and rounds
108+
identifies `kind: retained_observation / cached_state`, an indexed `artifact_ref`,
109+
the assessment's `json_pointer` inside that document, and
110+
`verification: verified / unverified`. A retained observation must agree with the
111+
committed record and its recorded digest. A readable cached assessment is used
112+
only when that original cannot be read consistently; its reference resolves to
113+
`state.json`, with `/final/assessment` or `/rounds/<index>/assessment`. Conflicting
114+
values retain an explicit gap instead of choosing the more favorable verdict.
115+
The index still records the missing or damaged observation and raw evidence.
116+
Observation verification does not certify all evidence: `delivery_accepted`
117+
separately requires current evidence and candidate binding to verify. Cached
118+
assessments always remain unverified, even if the other evidence is intact.
119+
Their raw measurements remain in the historical check for audit, but do not
120+
produce comparison metrics, derived gains or usage totals. Cached search scores
121+
remain labelled in the history and curve. Malformed or absent cached assessments
122+
cannot establish historical completion; cached timeouts stay incomplete.
123+
104124
Missing claim identities stay null with `identity_incomplete`. Existing INCONCLUSIVE
105125
is preserved; missing assessments are “Not assessed,” never fabricated INCONCLUSIVE.
106126
`improved` also requires a non-baseline selection, passing verified final and a gain
@@ -127,7 +147,7 @@ remain null without accounting observations; elapsed wall time is insufficient.
127147

128148
| Input | Authoritative facts | Deliberate gaps |
129149
| --- | --- | --- |
130-
| Native project / Benchmark | Plan, campaign, state, digest-bound observations/execution/evidence and retained code | Missing identities stay unknown; summaries cannot substitute for final |
150+
| Native project / Benchmark | Plan, campaign, state, digest-bound observations/execution/evidence and retained code; labelled cached history if observations are lost | Cached history cannot verify delivery; summaries cannot substitute for final |
131151
| Historical version 1 `summary.json` | Recorded final assessment and reported completion | No reconstructed rounds, independent final, binding or usage coverage |
132152
| Legacy run directory | `run.lock.json`, `status.json`, `evaluation.json`; paired single-experiment median | No invented iterative rounds/final or generalized L4 claim |
133153
| Apps `result.json` | Validated AppsResultV1, explicitly selected primary claim, fixed refs, all original claims | Operation success is not campaign completion; storage is not implicitly mounted/fetched |

‎src/inferrsi/apps/delivery/projection.py‎

Lines changed: 52 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -263,6 +263,7 @@ def iterative(snapshot, result):
263263
final_refs = []
264264
final_candidate = None
265265
final_observation = None
266+
final_assessment_verified = False
266267
result["rounds"], result["metrics"], result["checks"] = [], [], []
267268
records = [*rounds, *([final] if final else [])]
268269
for index, record in enumerate(records):
@@ -273,12 +274,25 @@ def iterative(snapshot, result):
273274
directory + "/observation.json", "assessment", record.get("observation_sha256")
274275
)
275276
# Compare cached assessment with its independently retained original.
276-
if observation and observation != {
277+
if observation is not None and observation != {
277278
k: v for k, v in record.items() if k != "observation_sha256"
278279
}:
279280
observation = None
280281
gap(result, "ASSESSMENT_CONFLICT", ref, "Cached and retained observations disagree")
281-
value = observation.get("assessment") if observation else None
282+
# Committed state retains history when the original is lost or conflicts;
283+
# this fallback is never current verification.
284+
historical = observation if observation is not None else record
285+
assessment_ref = ref if observation is not None else state_ref
286+
provenance = {
287+
"kind": "retained_observation" if observation is not None else "cached_state",
288+
"artifact_ref": assessment_ref,
289+
"json_pointer": (
290+
"" if observation is not None else "/final" if is_final else f"/rounds/{index}"
291+
)
292+
+ "/assessment",
293+
"verification": "verified" if observation is not None else "unverified",
294+
}
295+
value = historical.get("assessment")
282296
if (
283297
not isinstance(value, dict)
284298
or value.get("status")
@@ -290,7 +304,16 @@ def iterative(snapshot, result):
290304
result,
291305
"ASSESSMENT_UNAVAILABLE",
292306
ref,
293-
"The original assessment is absent or malformed",
307+
"No readable assessment remains in the retained observation or committed state",
308+
)
309+
elif observation is None:
310+
gap(
311+
result,
312+
"CACHED_ASSESSMENT_UNVERIFIED",
313+
ref,
314+
"Cached historical assessment from state.json is retained for audit; "
315+
"the original observation is unavailable or conflicts. Current verification "
316+
"is not established by this cached record.",
294317
)
295318
candidate = (
296319
snapshot.candidate(
@@ -317,7 +340,8 @@ def iterative(snapshot, result):
317340
parsed.append(usage_record)
318341
metric_start = len(result["metrics"])
319342
if value:
320-
result["metrics"].extend(_metrics(value, plan, phase, ref))
343+
if observation is not None:
344+
result["metrics"].extend(_metrics(value, plan, phase, ref))
321345
result["checks"].append(
322346
{
323347
"name": "frozen_assessment",
@@ -327,22 +351,32 @@ def iterative(snapshot, result):
327351
"gate_ref": plan_ref,
328352
"observation": value,
329353
"verdict": value.get("verdict"),
330-
"evidence_refs": [ref, *evidence_refs],
354+
"assessment_provenance": provenance,
355+
"evidence_refs": list(dict.fromkeys([assessment_ref, ref, *evidence_refs])),
331356
}
332357
)
333358
if is_final:
334359
final_value, final_refs = value, [ref, *evidence_refs]
335-
final_candidate, final_observation = candidate, observation
360+
final_candidate, final_observation = candidate, historical
361+
final_assessment_verified = observation is not None and value is not None
336362
result["final"] = {
337363
"attempt_id": directory.rsplit("/", 1)[-1],
338364
"fingerprint": record.get("candidate_sha256"),
339365
"protocol_ref": plan_ref,
340-
"assessment_ref": ref,
341-
"execution": observation.get("execution") if observation else None,
366+
"assessment_ref": assessment_ref,
367+
"assessment_provenance": provenance if value else None,
368+
"execution": historical.get("execution"),
342369
"status": (value or {}).get("status"),
343370
"candidate_binding": record.get("candidate_binding"),
344371
}
345-
assessment(result, value, ref, scope=plan.get("evaluation_settings", {}).get("scope"))
372+
assessment(
373+
result,
374+
value,
375+
assessment_ref,
376+
scope=plan.get("evaluation_settings", {}).get("scope"),
377+
)
378+
if value:
379+
result["acceptance"]["assessment_provenance"] = provenance
346380
else:
347381
best_after = record.get("best_after", {})
348382
result["rounds"].append(
@@ -352,8 +386,9 @@ def iterative(snapshot, result):
352386
"candidate_fingerprint": record.get("candidate_sha256"),
353387
"hypothesis": record.get("hypothesis"),
354388
"change_summary": record.get("change_summary"),
355-
"execution_state": (observation or {}).get("execution", {}).get("status"),
356-
"assessment_ref": ref,
389+
"execution_state": historical.get("execution", {}).get("status"),
390+
"assessment_ref": assessment_ref,
391+
"assessment_provenance": provenance if value else None,
357392
"verdict": (value or {}).get("verdict"),
358393
"reason": (value or {}).get("reason"),
359394
"score": (value or {}).get("score"),
@@ -415,6 +450,11 @@ def iterative(snapshot, result):
415450
if finished
416451
else "Required rounds or independent final assessment are incomplete",
417452
}
453+
if finished and not final_assessment_verified:
454+
result["completion"]["reason"] = (
455+
"Required search and independent final assessment completed according to "
456+
"cached state; unverified against the retained final observation"
457+
)
418458
if not final:
419459
gap(result, "FINAL_NOT_RUN", "final_acceptance", "Independent final evaluation is absent")
420460
fingerprint_bound = bool(
@@ -455,6 +495,7 @@ def iterative(snapshot, result):
455495
)
456496
current = (
457497
bound
498+
and final_assessment_verified
458499
and snapshot.verified(frozen_refs)
459500
and snapshot.verified(final_refs)
460501
and baseline["verified"]

‎src/inferrsi/apps/delivery/render.py‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,12 @@ def _markdown(value):
2020
return "".join(f"&#{ord(c)};" if c in '\\`*_{}[]()#+-.!|<>"&\n\r' else c for c in _text(value))
2121

2222

23+
def _assessment_source(value):
24+
if not value:
25+
return None
26+
return "cached state; unverified" if value["kind"] == "cached_state" else "retained observation"
27+
28+
2329
def _sections(result):
2430
acceptance = result["acceptance"]
2531
selected = result["selected_candidate"]
@@ -35,6 +41,8 @@ def _sections(result):
3541
["Completion reason", result["completion"]["reason"]],
3642
["Outcome", result["outcome"]["kind"]],
3743
["Original assessment", verdict],
44+
["Assessment source", _assessment_source(acceptance.get("assessment_provenance"))],
45+
["Assessment reference", acceptance["assessment_ref"]],
3846
["Acceptance scope", acceptance["scope"]],
3947
["Delivery acceptance verified", acceptance["delivery_accepted"]],
4048
[
@@ -101,6 +109,7 @@ def _sections(result):
101109
"Change",
102110
"State",
103111
"Verdict",
112+
"Assessment source",
104113
"Score",
105114
"Best score",
106115
"Selection / reason",
@@ -113,6 +122,7 @@ def _sections(result):
113122
r.get("change_summary"),
114123
r.get("execution_state"),
115124
r.get("verdict"),
125+
_assessment_source(r.get("assessment_provenance")),
116126
r.get("score"),
117127
r.get("best_score"),
118128
[r.get("selection_decision"), r.get("reason")],
@@ -197,6 +207,11 @@ def _curve(result):
197207
"Hover a point for its recorded value. Final acceptance remains separate.</figcaption>"
198208
"</figure>",
199209
]
210+
if any((r.get("assessment_provenance") or {}).get("kind") == "cached_state" for r in rounds):
211+
nodes.append(
212+
"<p>Some search scores are from cached state; unverified. "
213+
"See each round's assessment source below.</p>"
214+
)
200215
return "".join(nodes)
201216

202217

@@ -205,9 +220,12 @@ def reports(result, result_digest):
205220
f"Export: {result['export']['id']} · Snapshot: {result['export']['snapshot_at']} · "
206221
f"Kind: {result['export']['kind']} · Structured result SHA-256: {result_digest}"
207222
)
223+
verdict = result["acceptance"]["verdict"] or "Not assessed"
224+
if (result["acceptance"].get("assessment_provenance") or {}).get("kind") == "cached_state":
225+
verdict += " (cached state; unverified)"
208226
conclusion = (
209227
f"{result['completion']['status']}: {result['outcome']['kind']}. "
210-
f"Original assessment: {result['acceptance']['verdict'] or 'Not assessed'}. "
228+
f"Original assessment: {verdict}. "
211229
f"{result['completion']['reason']}."
212230
)
213231
md = [

‎src/inferrsi/apps/delivery/schema.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,7 @@ def empty_result(export_id, observed_at, version):
111111
"claim_id": None,
112112
"scope": None,
113113
"assessment_ref": None,
114+
"assessment_provenance": None,
114115
"delivery_accepted": False,
115116
},
116117
"metrics": None,
@@ -171,6 +172,10 @@ def validate_result(value):
171172
or value["completion"]["status"] != "completed"
172173
or not value["final"]
173174
or value["final"].get("binding", {}).get("verified") is not True
175+
or (
176+
acceptance.get("assessment_provenance") is not None
177+
and acceptance["assessment_provenance"].get("verification") != "verified"
178+
)
174179
):
175180
raise ValueError("Delivery acceptance requires a completed final with verified binding")
176181
for measure in value["usage"].values():

0 commit comments

Comments
 (0)