Skip to content

Commit ff93302

Browse files
authored
perf: reduce knowledge retrieval manifest work with measured evidence (#193)
Independent source/evidence LGTM: #193 (review) Current-main integration delta: #193 (review) Ready full-check/installed-package both Python versions passed: https://github.com/InferMatrixLab/InferRSI/actions/runs/40500855160 Required policy refreshed success: https://github.com/InferMatrixLab/InferRSI/actions/runs/40502162737 Frozen CPU synthetic-corpus measurement applies only to pinned baseline/candidate, not later catalog revisions or model/hardware efficacy. Merge does not grant #190 goal acceptance.
1 parent ba7ba1d commit ff93302

42 files changed

Lines changed: 63852 additions & 7 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/knowledge-retrieval-audit.md‎

Lines changed: 266 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Knowledge retrieval implementation plan
2+
3+
**Goal:** Reduce repeated work in `Bundle.manifest()` while retaining the same
4+
validated manifest, revision and retrieval results.
5+
6+
**Architecture:** Reuse resource digests and validated content paths within one
7+
manifest call. `Bundle.metadata`, `Bundle.content` and `Bundle.dependencies`
8+
remain mutable, so no digest or manifest survives that call. Keep the existing
9+
package snapshot, checkout and store ownership boundaries.
10+
11+
**Scope:** `src/inferrsi/knowledge_operations.py`, focused regression tests in
12+
`tests/test_bundle_manifest.py`, and the source-grounded audit in
13+
`docs/knowledge-retrieval-audit.md`. Catalog/source bytes, scoring, query
14+
semantics, ordering, budgets, omissions and dependency closure are unchanged.
15+
Performance gates, revision pins and frozen measurement inputs belong to the
16+
governing acceptance issue and its evidence packet, not reusable defaults.
17+
18+
## Plan -> code -> test -> review
19+
20+
1. Inspect the reader, selection helper, store and existing tests. Compare the
21+
alternatives: local digest reuse is bounded; persistent Bundle caching is
22+
unsafe with mutable inputs; changing selection is outside this slice.
23+
2. Add a small synthetic Bundle fixture with file and directory dependencies.
24+
Verify the complete manifest and revision against the existing digest
25+
formula; count hashing calls and verify each resource is hashed once per
26+
call. Verify repeated calls detect changed bytes, removed resources, unsafe
27+
paths, changed metadata/requirements and changed dependency revisions.
28+
3. In `Bundle.manifest()`, build one sorted resource digest map after content
29+
validation and required-resource checks. Reuse it for dependency validation
30+
and the returned resource inventory. Reuse validated required paths while
31+
preserving missing/unsafe-path validation and existing error categories.
32+
4. Run `python -m pytest tests/test_bundle_manifest.py
33+
tests/test_knowledge_operations.py tests/test_collect_selection.py
34+
tests/test_knowledge_store.py` and `python -m ruff check
35+
src/inferrsi/knowledge_operations.py tests/test_bundle_manifest.py`.
36+
The accountable parent runs the full suite, installed-package checks and
37+
paired performance/quality protocol after its independent fixture freeze.
38+
5. Review the diff for persistent caching, changed resource closure, digest
39+
order, invalid-input behavior, private data and unrelated changes. Record
40+
actual checks separately from pending measurements. Commit with the actual
41+
configured identity, DCO sign-off and material-agent attribution; hand the
42+
revision to the accountable parent for independent review and integration.
43+
44+
The audit distinguishes the delivered optional whole-resource context from
45+
historical efficacy claims. It does not reinterpret the prior campaign or
46+
claim that less hashing improves selection quality or agent outcomes.
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# CPU knowledge retrieval evidence
2+
3+
Acceptance scope and ownership: [InferRSI #190](https://github.com/InferMatrixLab/InferRSI/issues/190).
4+
This packet tests a manifest implementation that preserves retrieval behavior.
5+
The frozen quality/parity and paired CPU performance gates passed. Campaign
6+
acceptance, merge and issue closure remain with the authorized owner.
7+
8+
| Segment | Contents |
9+
| --- | --- |
10+
| `plan/` | Prospective protocol, current freeze, preserved first freeze and pre-measurement review amendment |
11+
| `validator/` | Standard-library runner, 16 development and 8 withheld synthetic cases with source-grounded partial relevance labels |
12+
| `results/` | Complete quality comparison, paired raw API/CLI measurements, baseline profile, #129 delivery trace, editable EN/ZH architecture artifacts and qualified local software/environment summaries |
13+
| `patch/` | Fixed runtime candidate and exact source identity |
14+
15+
The current prospective freeze is
16+
`a465367319f3058398eb1f8579ee5d4174b8cb90457793e90788fb5796151057`.
17+
Independent review corrected completeness checks and removable assertions before
18+
formal candidate measurement; `plan/amendment-1.md` records the change. The
19+
withheld split was released only after the runtime candidate was fixed. Fixture
20+
text and labels did not change between freezes. Root `.gitattributes` preserves
21+
evidence bytes; copying or committing must retain the recorded hashes.
22+
23+
The full quality result has **96 outputs per revision, exact parity and unchanged
24+
source/corpus identities**. For 20 labelled answerable cases, search/default
25+
collect score Success@5/MRR@5 = 0.20/0.20; relevance/context-item ranking scores
26+
1.00/0.9625. Four labelled negatives return empty items in 4/4 search/default
27+
cases and 2/4 relevance/context cases. These are unchanged existing #129
28+
behaviors, not a quality improvement from this runtime patch.
29+
30+
Context-item ranking does not establish correct body delivery. A post-hoc
31+
inspection found the labelled owner's context delivered in **19/20** positive
32+
cases. `withheld06` finds its correct item first but requires 170,782 context
33+
bytes against a 65,536-byte budget, so that owner's context is omitted. This
34+
diagnostic does not change the frozen quality gate. Synthetic, AI-authored,
35+
partial source labels are not human gold judgements or complete Recall; no
36+
agent-task efficacy or production-distribution claim follows.
37+
38+
The 20-pair CPU result observed **40.01%** hot-relevance paired-median latency
39+
reduction, 95% interval **[38.38%, 41.06%]**, with all 400 output pairs and
40+
80 CLI stdout pairs equal. [performance.md](results/performance.md) retains
41+
all per-mode/query p50/p95 rows, including search regressions and CLI relevance
42+
decode p95 **1.810 → 3.046 ms**. Independent read-only recomputation passed;
43+
reviewer model/effort is unknown and no human/maintainer acceptance is recorded.
44+
45+
See the [audit](../../docs/knowledge-retrieval-audit.md) for source contracts,
46+
per-query examples, failure qualifications and follow-up priorities. Local
47+
software verification includes a failed whole-suite run whose initial failures
48+
were reproduced on baseline, successful focused manifest checks and successful
49+
isolated wheel/sdist construction; it must not be described as a green full
50+
suite. No model, GPU or inference-service benchmark ran.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Runtime patch boundary
2+
3+
The evaluation-material author changed no runtime source. `candidate.json`
4+
records the independently authored runtime commit and exact Python-source
5+
digest used by the formal comparison. The implementation reuses digest/path
6+
work within a manifest call; it introduces no cross-call mutable cache.
7+
8+
Runtime integrity tests and independent review belong beside those revision
9+
records. Exact-output parity does not replace mutation or unsafe-path regression
10+
checks. Later evidence/documentation integration must preserve the tested
11+
runtime bytes; merge and issue closure remain with the authorized owner.
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"schema_version": 1,
3+
"governing_issue": "https://github.com/InferMatrixLab/InferRSI/issues/190",
4+
"baseline_sha": "4817f92121df73fd9cd5fb4a319e3c56a68b4a5b",
5+
"candidate_runtime_sha": "407bb244651bec1c68f4b3e94b0c697b233c4a09",
6+
"runtime_author_commit": "864fd74e709a1411e4f8dfd3051cc9b737d8c333",
7+
"candidate_python_sources_sha256": "d6edbaf975cf3bc00902d6aa7cf2bb955dc58b3e37c74f59bb226c5f0d2fa261",
8+
"scope": "Reuse sorted resource digests and validated paths within each Bundle.manifest call. No cache survives a call and retrieval outputs must remain identical.",
9+
"runtime_source": "src/inferrsi/knowledge_operations.py",
10+
"runtime_regressions": "tests/test_bundle_manifest.py",
11+
"integration_note": "Evidence, audit, diagrams and read-only inventory registration are integrated separately; compare their packaged runtime source to this fixed tested candidate. No merge or issue closure is implied."
12+
}
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Frozen evaluation protocol
2+
3+
Governing acceptance: [InferRSI #190](https://github.com/InferMatrixLab/InferRSI/issues/190).
4+
The machine-readable protocol is `protocol.json`; the freeze manifest commits
5+
protocol, independently constructed inputs and validator bytes before candidate
6+
measurement. This packet is authored by Codex `gpt-6.1-sol` / `xhigh`.
7+
8+
1. Read the shipped source content, construct 16 development and 8 withheld cases,
9+
and label defensible relevant package identities by original resource paths.
10+
Do not consult rankings while labelling. The partial labels are sufficient
11+
for Success@5/MRR@5, not complete Recall. The queries are synthetic and have
12+
no production-distribution claim.
13+
2. Validate fixture references and freeze all authored inputs and scripts. Keep
14+
withheld queries/labels outside the runtime author's and orchestrator's view
15+
until the candidate revision and source digest are fixed.
16+
3. Profile the read-only baseline separately. Run quality/parity and the frozen
17+
20 balanced paired groups with one interpreter and an equal complete corpus.
18+
Each process explicitly imports the selected checkout rather than a possibly
19+
unrelated editable install. Keep all raw timings and exact-output digests.
20+
4. Report search and relevance retrieval separately at identical API scope
21+
(`repo=inferrsi`, `limit=5`). Report default collect, context delivery and CLI
22+
startup separately. The current collect CLI lacks repo/backend filters;
23+
its wider-scope startup cost is not an equivalent retrieval-quality result.
24+
5. Apply frozen gates, self-check output integrity and submit artifacts for an
25+
independent review. This evaluation does not establish model task efficacy,
26+
full-library recall, service throughput or OS filesystem-cache-cold latency.
27+
28+
The primary performance statistic is the median of 20 paired reductions, each
29+
formed from the median latency of four fixed hot relevance queries. The 95%
30+
percentile bootstrap resamples complete paired groups with seed `19020261027`
31+
for 10,000 iterations. The prospective issue bar requires at least 20% median
32+
reduction and a lower confidence bound above zero. No trimming or post-hoc
33+
choice of the fastest query, summary or run is allowed.
34+
35+
Exact parity covers the complete canonical serialization, with readable
36+
projections for identity order, revisions, resource-hash inventories, omissions,
37+
collision metadata, material bytes and context bytes/budget. The API timer
38+
excludes the evaluator's serialization and hashing work. Full CLI timings
39+
include process creation, imports, command work and stdout capture; decoding is
40+
measured separately. Fresh-reader API calls include reader construction;
41+
reused-reader calls follow one untimed pass across all four hot queries.
42+
43+
No candidate has been timed while this protocol is being authored. Missing or
44+
incomplete evidence remains INCONCLUSIVE. Existing #128/#129 evidence stays
45+
historical and is not relabelled as this campaign's measured result.
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Pre-measurement amendment 1
2+
3+
Independent review found two fail-closed defects after the first packet freeze
4+
and before any formal candidate evaluation or timing:
5+
6+
- Positional `zip` comparisons could silently ignore missing output rows. A
7+
synthetic candidate with incomplete API/CLI measurements could satisfy the
8+
timing gate without complete evidence.
9+
- Fixture and warmup checks used Python assertions, which optimized Python could
10+
disable.
11+
12+
The validator now requires the exact case-by-mode sets once per side, the full
13+
pair-ID range and each prescribed AB/BA order. Missing, extra or duplicate rows
14+
or pair IDs make the comparison INCONCLUSIVE and prevent a performance PASS.
15+
Quality comparison explicitly checks its complete expected set, including all
16+
96 rows per side for the full 24-case run. Keyed comparisons replace `zip`.
17+
Critical checks raise explicit exceptions; child processes set PYTHONOPTIMIZE=0
18+
and record optimization flags. Synthetic selfchecks cover incomplete/duplicate
19+
quality rows, missing benchmark rows and duplicate paired groups.
20+
21+
The original `plan/freeze-v1.json` is preserved with content digest
22+
`2a90de774129b06597c3c99e5b06d05bf735026829deb84376d06a985cdcc276`.
23+
Before this supersession, only source-label/static validation and
24+
baseline-v-baseline mechanics had run, never formal candidate measurement.
25+
The revised `plan/freeze.json`
26+
supersedes it prospectively. Fixture text, relevance labels, scoring rules,
27+
query workloads, numeric gates and bootstrap method/seed are unchanged. The
28+
runtime candidate was already fixed before the withheld fixture was released.
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"schema_version": 1,
3+
"frozen_at_utc": "2026-10-27T11:17:10.632059+00:00",
4+
"files": {
5+
"plan/protocol.json": "dc0fbbb4438a072fe1dbc530f5a60a4b1e1e779cc32697fd9adccc0eac3bc4fc",
6+
"plan/README.md": "12e0bb007b6a4ce7a4177e1843ae46ab3327fdd7fcc6591c3332db5684f83c3d",
7+
"validator/development.json": "d2f428c41787bfa6279e8a0b56532e76554482ab98dca553e7e9febe287e8dfb",
8+
"validator/withheld.json": "5bf9b8e55871a986cf7848959bf83e401821939acd13ee067563dc83952182e9",
9+
"validator/evaluate.py": "47d36dd7b0a0967cb74ae53a31646b8bf063958b3845769447c6e8af8f42e5e1",
10+
"validator/README.md": "2ca7c11a3add1a277c7c60dc24820d8fe5077e239d5deaf4ea6b0fe1195abe7c"
11+
},
12+
"content_sha256": "2a90de774129b06597c3c99e5b06d05bf735026829deb84376d06a985cdcc276"
13+
}
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
{
2+
"schema_version": 1,
3+
"revision": 2,
4+
"supersedes_content_sha256": "2a90de774129b06597c3c99e5b06d05bf735026829deb84376d06a985cdcc276",
5+
"frozen_at_utc": "2026-10-27T11:22:45.564236+00:00",
6+
"files": {
7+
"plan/protocol.json": "91f2966ddf7d37683cd2a79cd995ec1ad7b41993ea1e6c2cd008db48a3b99351",
8+
"plan/README.md": "12e0bb007b6a4ce7a4177e1843ae46ab3327fdd7fcc6591c3332db5684f83c3d",
9+
"validator/development.json": "d2f428c41787bfa6279e8a0b56532e76554482ab98dca553e7e9febe287e8dfb",
10+
"validator/withheld.json": "5bf9b8e55871a986cf7848959bf83e401821939acd13ee067563dc83952182e9",
11+
"validator/evaluate.py": "e5b8c99f6955420e67bb0d9fbcc4c08a6c8a95d857385a5d9f0a0f2f74e0f194",
12+
"validator/README.md": "e9b5f55e2b74b184a6bf2d31e342074e120663fa48286d6b14496491bc39eea4",
13+
"plan/amendment-1.md": "4feac787c0fff35bab49b04f1dc2fdd7648febc25a770a59d359f7d7ddb30c2c",
14+
"plan/freeze-v1.json": "a1f727ea4914b194469e5183cc2df6c7ec2b42614cc294dc14c0ec0fbd8dcb05"
15+
},
16+
"content_sha256": "a465367319f3058398eb1f8579ee5d4174b8cb90457793e90788fb5796151057"
17+
}

0 commit comments

Comments
 (0)