Skip to content

feat(knowledge): unify search across cases, skills, and records - #466

Merged
david6666666 merged 16 commits into
InferMatrixLab:mainfrom
jiaran-king:codex/knowledge-find-pr
Nov 17, 2026
Merged

david6666666 merged 16 commits into
InferMatrixLab:mainfrom
jiaran-king:codex/knowledge-find-pr

Conversation

@jiaran-king

@jiaran-king jiaran-king commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Scope

Refs #379

Agents can describe an optimization problem, find relevant guides, records, and historical cases in one query, and open the source text with its provenance, applicability, limitations, and revision. This PR delivers that local search and read workflow.

Previous search behavior

Entry point Behavior and impact
knowledge search / knowledge research Search or expand material from the installed record catalog. Extra words in natural-language queries can prevent a match in entry points that require every term to match.
collect --relevance Ranks packaged skills. Finding checkout cases requires selecting a different source and kind.
collect / read Collect or read by source, ID, or revision. Large bundle inventories can take up much of the tool response delivered to the agent.

In earlier replays, some relevant cases fell outside the selected results. Some selected material was cut off in the outer tool response. Finding a match and delivering its text are separate steps.

Changes

  • inferrsi find "<problem description>" uses a shared term-based ranking for registered cases, skills, and packaged records. It weights titles, summaries, tags, and findings more heavily, precomputes term frequencies, and gives long bodies less weight. It supports word boundaries, common separators, Chinese and English aliases, and the --kinds filter.
  • Re-derive 35 structured findings from the original text of seven cases so searches can match the direction tried, outcome, reasoning, and conditions for revisiting it. Each preview shows the applicability or bounds recorded in the source.
  • Return a source-qualified ID and an executable open command, with open_argv in JSON. The search uses the same reader to check that a hit can be opened. read --material --omit-resources returns the text, sources, and revision. A full read includes the resource inventory.
  • Use find → read as the default discovery workflow in the README, operations guide, and plugin instructions. The existing exact lookup, read, and resource commands remain available to callers. The Codex and WorkSwarm plugins read material through their respective installed launchers.
inferrsi find "cache per-step masks ran out of device memory" --kinds case --limit 1

This query returns the Qwen Ascend case and its open command. The read response includes the revision for later citations pinned to that version.

Review follow-up

  • Deduplicate query terms after alias expansion. Record bodies exclude the title, summary, tags, and applicability fields that already contribute to ranking.
  • Read the packaged record's verification field and structured verification limitations. The preview marks unverified material in both JSON and plain text.
  • Report the indexed library in source_location. An installed Skill or Case opens its original body through the checkout: reader route with the installed root.
  • Parse Skill descriptions with the shared YAML reader when available. Base installations use the text fallback, and --kinds record skips the unused skill root.
  • Write the exact LF bytes in the Windows-sensitive test fixture so the read-back assertion is independent of the host line ending.
  • Ignore common English function and question words during scoring. Terms such as not, without, before, and after still contribute. When the original query explicitly names one supported backend, results registered for a different backend receive half their lexical score.
  • Use topic aliases for lexical matching. 图模式 can match cuda graph, and the explicit 昇腾 in 昇腾 图模式 gives Ascend results priority over otherwise equal CUDA results. Both cases have regression coverage. David's review
  • Restore the path-only _indexed_directories interface required by the Knowledge migration contract. The existing reader and find share the validated index loader.
  • Lead with find → read in the WorkSwarm entry skill and document the packaged Skill fallback. The Codex entry skill now describes when find returns no_match. Steward re-review

Search replay results

The kind-labeled sets check whether the expected kind appears in the top three.

Query set and criterion Before the findings rewrite 8c0a7bfc 51aad393 ce5c8865
#380, expected kind in the top three 16/17 15/17, including 11/11 cases 15/17, including 11/11 cases 15/17, including 11/11 cases
Evidence-based kind labels, expected kind in the top three 21/34 24/34, including 7/10 cases 22/34, including 4/10 cases 22/34, including 3/10 cases
Nine frozen development queries, target case ID in the mixed-search top three — 7/9 7/9 7/9
Same queries restricted to cases, target ranked first and full text read back — 9/9 9/9 9/9

On 51aad393, the evidence-based labels registered three fewer Case-kind hits and one more Skill-kind hit than on 8c0a7bfc. Temporarily restoring repeated-term scoring on 51aad393 recovered all three Case-kind hits in a local diagnostic. The latest revision gained one Record-kind hit and lost one Case-kind hit. The lost hit was case.sarathi-chunk-budget on a Ref2VA delivery-latency query. The frozen queries check the exact target ID and full-text read-back.

Both CUDA graph capture and How should I warm up CUDA graph capture? now rank pytorch.cuda.compiler.cudagraph_shapes_and_lifetimes first. The unrelated skill-evolution-playbook drops out of the top three for the full-sentence query. The query about caching per-step masks ranks case.qwen-ascend-rejected-directions first. The question about word error rate under concurrent service ranks case.drama3-rejected-directions first.

The frozen set includes five Qwen queries and four about shared-resource hashing. The returned text matched the frozen source byte for byte and included the same provenance, applicability, and limitations. Reopening at the returned revision gave the same result. The follow-up c3364c25 reproduced the figures in the ce5c8865 column.

On 8c0a7bfc, the JSON response for package:inferrsi.common.skills.optimization_search_policy went from 223,782 bytes for a full read to 23,757 bytes for a compact read. The text, provenance, applicability, and revision matched.

Qwen-Image retrieval comparison

This section records the earlier agent-run comparison. The current review revision was evaluated with the fixed-query replays above.

Two fresh GPT-6 Luna / max subagent sessions with independent contexts ran the scaled qwen21-denoise-npu task package from #338. They ran in order A then B on the same Ascend 910B2C, comparing the previous session knowledge entry point with find → read. Both sessions had access to the knowledge base and could use historical material returned by their entry point. The corpus, delivery method, and agent-written queries could differ. Each session completed eight distinct candidates and an independent final run with pairs=2, samples=3, warmups=1. The benchmark used five correctness cases, three of which were timed.

Metric A — Previous retrieval B — find → read
Distinct candidate rounds and final task correctness 8/8, PASS 8/8, PASS
Selected candidate Round 8 Round 8
Independent final speedup 1.2061× 1.1859×
Campaign wall time 1,262.1 s 1,229.1 s
Total tokens for the execution session 6,205,904 4,308,099
Approximate token increase during the campaign 4,579,383 2,881,139
Input tokens 6,135,780 4,247,352
Cached input tokens 5,969,408 4,075,776
Uncached input tokens 166,372 171,576
Output tokens 70,124 60,747
Uncached input plus output tokens 236,496 232,323
Queries / returned items / unique full-text reads / items used in decisions / fallbacks 3 / 15 / 2 / 0 / 0 3 / 15 / 4 / 2 / 0

Returned items are counted across queries, including repeat hits. Full-text reads and uses count distinct items. A received full text three times across two items. B opened full text six times across four items. Use is counted where the agent recorded a link from the material to a candidate or a decision about which direction to try. Cached input is included in total input. The campaign token estimate follows #338 and uses cumulative host counts sampled before the start and near final completion. Uncached input plus output and the campaign estimate are usage figures. Billing requires separate accounting.

B recorded more uses of retrieved material and had lower campaign wall time and total session tokens. A's final candidate had lower median latency in all three timed cases. Each retrieval method ran in one agent session, with three queries per session. This comparison was selected after observing the results, and its conclusions apply to these two sessions.

Final results by case, round-by-round results, and retrieved material

Each final run paired the baseline and candidate measurements within the same arm for every case.

Arm Case Baseline median (ms) Candidate median (ms) Speedup
A compact 12.781 10.557 1.2107×
A portrait 20.966 17.493 1.1985×
A landscape 29.714 24.577 1.2090×
B compact 12.695 10.686 1.1880×
B portrait 20.748 17.545 1.1826×
B landscape 29.517 24.865 1.1871×

Cumulative tokens are estimated from host counts sampled near the end of each round, minus the count before the start. These samples give approximate usage by round. The campaign retained the other candidates and their feedback.

Round A change A speedup A cumulative tokens B change B speedup B cumulative tokens
1 Reuse text projections 1.0280× 931,628 Reuse text projections 1.0265× 587,595
2 Reuse RoPE sine and cosine 1.0532× 1,576,687 Reuse prefix timestep modulation 1.0236× 717,006
3 Fuse gate/up projections 1.0525× 1,882,004 Select decode modulation rows directly 1.1397× 860,066
4 Broadcast decode modulation 1.1796× 2,325,210 Reuse key masks 1.1338× 1,014,097
5 Preconvert Q/K norm weights 1.1852× 2,804,046 Reuse RoPE angles 1.1576× 1,675,210
6 Reuse modulation splits across layers 1.1671× 3,187,556 Fuse gate/up projections 1.1441× 2,230,740
7 Reuse K/V buffers 1.2062× 3,723,445 Batch timestep modulation 1.1621× 2,479,247
8 Reuse attention masks 1.2457× 4,287,219 Reuse RoPE sine and cosine 1.1805× 2,744,097
Final Independently retest round 8 1.2061× 4,579,383 Independently retest round 8 1.1859× 2,881,139

Full-text material delivered or read during the sessions

Arm Material ID / revision Bytes Recorded influence
A package:inferrsi.common.skills.skill_evolution_playbook
sha256:fdba8ee31bc48d8a3cec0d5da3200324432b2b1782c3199c1de163042ec586d5
9,448 No recorded use in a decision
A package:inferrsi.ascend.skills.ascend_operator_tuning
sha256:54f3556be2fbd4b5e6e2af68c7b4df9a7c1d2c53fde23150132063d20a25f267
12,236 No recorded use in a decision
B vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
sha256:b1740d8dd28f3c2f7ec2e4fbb9233a7ce311f52210fa3c6365bb320ad191e142
4,696 Informed the RoPE angle caching scope in round 5
B case.qwen-ascend-rejected-directions
sha256:039a074c881a924327549416d143fcf7cb550493e87352113b45a08d8717827a
2,564 Informed decisions on attention fusion and caching expanded masks
B case.qwen-b300-rejected-directions
sha256:0a9a9890243d4fd8a58989fbf7e27cab65cacc60f2471892da00ffc3b19e49d6
2,656 No recorded use in a decision
B torch-npu
sha256:dab34d1dcd1fe61b1fd1223d592a3d1ad2d878ae5d3f6d4edf8d1ec29487462f
4,948 No recorded use in a decision

The historical entries were available as leads for the experiments. The counts reflect what the sessions attributed to those entries. Causal effects require separate evidence.

The runs used pairs=2, samples=3, warmups=1 for every candidate and final, covering five correctness cases and timing three. The exact wait for the initial device allocation is unknown. Each session kept the same device across rounds after allocation. Campaign wall time includes agent reasoning, retrieval, editing, and evaluation. The benchmark scope is scaled synthetic denoising. Real-weight image quality and end-to-end service performance would require separate evidence. The original artifacts and full evidence tree are held privately.

Checks and evidence limits

  • On 33de913c, 43 plugin and runtime checks passed. In an extracted WorkSwarm archive with PyYAML absent, default find returned the Qwen case and its open_argv read the body, sources, and revision. The documented fallback fetched a packaged Skill whose body matched the source byte for byte. An unmatched query returned no_match.
  • On c3364c25, 263 affected tests passed, including the compatibility suite and both alias regressions. Two Windows-only checks were skipped on macOS. Ruff passed.
  • On ce5c8865, 240 affected tests passed locally. The Python 3.10 and 3.11 Ready full suites each reported one failure in the Knowledge migration compatibility test for _indexed_directories. Ready CI on ce5c8865
  • On 51aad393, the affected suites passed 151 tests. A clean Python 3.11 wheel returned the Qwen Case and the packaged fallback body with PyYAML absent. With PyYAML installed, it parsed the displayed description and opened the installed Skill's full body byte for byte. Ruff and the pinned index check passed.
  • On dd28a9df, the exact quickstart command ranked the Qwen case first and put the out-of-memory finding at the top of its preview. The returned read command opened the source text. The ranking and no-PyYAML fallback tests passed. All 80 generated index records passed the check against the pinned source revision.
  • In an earlier base Python 3.11 wheel check without PyYAML, find returned kernel-writing-guide for cuda graph warmup. collect --relevance started a fresh search of packaged material and returned inferrsi.ascend.skills.ascend_aclgraph_capture_replay. The guide tells callers to assess the new hit before reading its body via resource with its ref, skill_documents.original, and revision. The read-back for both review queries matched the wheel source byte for byte. The case find → read flow also worked in the base install. David's review
  • On 8c0a7bfc, the relevant knowledge search, read, index, installation, module boundary, and plugin suites had 286 passes and 4 skips. Two DSH checks failed with a Node SIGABRT. The same two checks failed in the same way on main at 891aadda. Ruff, diff, and generated index checks passed.
  • The wheel and sdist built with default version detection. Clean wheel installations passed on Python 3.10 and 3.11, with 929 resources matching the checkout byte for byte.
  • A kind hit means the expected kind appeared in the top three. Material coverage and differences in vocabulary still affect ranking.
  • The tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 requester and maintainer approved using find → read as the default discovery workflow for feat(knowledge): unify search across cases, skills, and records #466 in the README, quickstart, and plugins, with the existing commands available for compatibility and fallback. The task: 改进真实优化任务中的材料发现与使用 #380 decision held back the short guidance tested at the time. The Qwen comparison recorded use of retrieved material. Consistent improvements in decision quality and optimization gains remain to be established. Requester decision · Maintainer decision

Knowledge authorship declaration

The PR author confirmed that the earlier search implementation used Claude Opus 5.5 at max effort. A configuration mapping error caused the CLI to label it Opus 5.

{
  "schema_version": 1,
  "head_sha": "33de913c97be00e9d2dee61b9eec2109ba540935",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Claude Code (earlier search implementation)",
      "model_id": "claude-opus-5-5",
      "effort": "max",
      "role": "author"
    },
    {
      "tool": "Claude Code (current-main knowledge migration)",
      "model_id": "claude-opus-5-5",
      "effort": "max",
      "role": "author"
    },
    {
      "tool": "Codex (case findings rewrite, read handoff, regression fixes, and CLI integration)",
      "model_id": "gpt-6.1-sol",
      "effort": "max",
      "role": "author"
    }
  ]
}

…PyYAML

Each casebook entry already recorded the directions tried, their outcome and
when a retry would be justified, but only as prose or a Markdown table that
ranking could not weigh. Add a `findings` list to all seven cases: direction,
outcome, recorded reason, revisit condition, retrieval keywords and `scope`.
Every text field is taken from the existing case body; no claim, cause or
measurement is added and no existing field changes.

`scope` states what would have to change for an observation to stop holding:
`local` (that host or harness), `regime_bound` (the cohort the case names) or
`general` (a property of the method). It is not derivable from the outcome --
a rejection can be bound to one resolution or hold everywhere. Of 35 findings,
24 are regime_bound, 6 general and 5 local. The source-contract cases are
regime_bound: each reads one pinned upstream commit and says so in its own
applicability, and a later revision can change the dispatch it describes.

Reading a case parses no skill frontmatter, but importing the skill catalog
required PyYAML, so a base install could not discover or open one. The parser
is now imported where frontmatter is parsed. Its absence raises
MissingSkillsExtra, a ValueError subclass in a module of its own so catching it
never imports the parser, and `collect` lets it through rather than reporting
"nothing found" for an install problem. The snapshot tool relied on the old
import-time failure to stop before side effects, and now asks for the parser
explicitly at that point.

The reader exposes `cases()` and `case_resource_path()` so a caller ranking
cases resolves them by the same rule `read` uses.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=max] (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
…ndings

Mechanical follow-up. The generator pins each card's source hashes to a
revision, so the three skills that own the edited cases -- the optimization
casebook, the kernel-writing guide and the search policy -- held hashes for
the case files as they were before findings were added, and `indexed_skill`
rejected those projections as stale. Regenerated with
tools/build_skill_index.py against the previous commit.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=max] (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Port the verified find and read flow to the current Knowledge package and Apps-facing CLI. Keep query fixtures, compact exact reads, and standalone plugin coverage together.

Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Agent-assisted-by: Claude Code (author)
Agent-assisted-by: Codex (author)
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Skill evaluation

PLANNED: not executed; there are no task outcomes yet.
No approval, model efficacy or current-head test result is implied.

  • Next: run the pinned source/index --check below at head 33de913c97be00e9d2dee61b9eec2109ba540935.
  • Task selection: confirm the prepared task tests the changed claim.
  • Selected comparisons: 1 · Estimated task calls: 2. Uncovered/unsupported changes: 34; Canonical library paths: 6; Generated projection paths: 4; Other paths: 24. These are paths, not independent claims.
  • Return: command, input/source pin, exit status, outputs/failures and untested claims for independent verified/refuted/unverified disposition.
Author verification steps (nonpaid; not yet run)
  1. Bind the checkout. Run git rev-parse HEAD; it must equal 33de913c97be00e9d2dee61b9eec2109ba540935. If it differs, obtain a current plan; do not reuse old results.
  2. Check source/index consistency. At that PR checkout, using the reviewed repository tool and its documented development dependencies, run:
python tools/build_skill_index.py --source-revision 8288a004c333b0e338b3ad31fe823052a142b90a --check

The declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.

  • optimization-search-policy
    Selected by the trusted path mapping: src/inferrsi/knowledge/data/skill_library/methods/optimization-search-policy/cases/sarathi-chunk-budget.json.
    Task case-05: Diagnose this blocked request at the recorded decision boundary; propose the next read-only check.
    Scope: Synthetic finite-state observations; not a production trace or engine benchmark.
    Required answer fields: diagnosis, evidence, next_check, reuse_authorized.
    Independent check: frozen kv-admission-diagnostic-v1 grader/schema bound in details. Confirm it tests the changed claim, not merely the touched path.
    From the downloaded skill-eval-plan artifact directory, inspect the first prepared task and its required answer schema (read-only; not grading):
python -m json.tool optimization-search-policy/plan/tasks.json
python -m json.tool optimization-search-policy/plan/schema.json

Uncovered scope by location: Canonical library paths: 6; Generated projection paths: 4; Other paths: 24. Path counts are not claim counts; no metadata-only conclusion is inferred.
3. Return evidence for independent review: head, check/task/source pin, command, environment, exit status, outputs/failures and untested claims. Label scripted controls. Request each claim's disposition: verified, refuted or unverified; this plan proves neither model provenance nor benefit.
Check recipes and evidence limits

Evidence and logs · Retention: 30 days.
Retained artifacts are verifiable only before Unix time 1797488496 while the linked artifact is available. Expired evidence is unverified.

Plan bindings, budget, frozen exam and advisory details

Advisory: obtain missing evidence

Retained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Applicable only while these revisions and inputs match the current PR.
Paid execution requires trusted label authorization and cumulative PR reservation. Unknown accounting blocks new paid calls; planning is not execution.
Pipeline evidence only; offline fixed outputs do not establish skill efficacy.
Unchanged-correct is valid neutral evidence, not incremental benefit.
Content/source review and authorship admission remain separate.
Trusted regrading checks retained answers, not model/hardware provenance or a hostile executor; this is not an efficacy certification.

  • Retained evidence is planned / unexecuted.
  • Uncovered/unsupported scope needs a case or explicit reviewer disposition.
    Plan SHA-256: 7846444e35979d6df30bc164eb1506d59b661e3be0250d7c2b2208b1ebb9a09e
    Base: 891aadda6d4abc050def2baa2814b284f87808e5 · Head: 33de913c97be00e9d2dee61b9eec2109ba540935
    Trusted implementation: 1acfacf8d49a13edf8d1fbdfde224c11ddbee8bf

Model/settings: {"effort":"not_applicable","name":"claude-opus-5-5"}
Budget: {"comparisons":1,"estimated_calls":2,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":200000},"provider_config_sha256":"b28d80f8194fbe0f5a9f01515efb24b218979fd45f702776560da276f4b30d04","reservation":{"amount":"20.34","currency":"USD","group":"default","limit":"200","per_request":"10.17","pricing_sha256":"2a82453ec18b75b92e12f9a9c18e6b57811a3ef593c31e4639b839df218cac39","requests":2}}

Skill Comparison Cases Estimated calls
optimization-search-policy update case-05 (holdout) 2

Frozen exam hashes: {"cases":"a0dd4e1615402a3a629f925c15435470095c27eeac54c4a2080c2d874abeb031","grader":"e25cfee943052aaefbddf3f545fe6d56d95ab1edb9723cd9353feeabc954ea9a","schema":"46ed2e63dadd5f9fa60107ff58b020554b42f47472ca80b4de9add79d92f8449"}

Selection/task metadata (not a relevance verdict):

{"adapter":"kv-admission-diagnostic-v1","basis":"trusted_path_mapping","matched_paths":["src/inferrsi/knowledge/data/skill_library/methods/optimization-search-policy/cases/sarathi-chunk-budget.json"],"required_answer_fields":["diagnosis","evidence","next_check","reuse_authorized"],"tasks":[{"id":"case-05","provenance":"Synthetic finite-state observations; not a production trace or engine benchmark.","request":"Diagnose this blocked request at the recorded decision boundary; propose the next read-only check.","split":"holdout"}]}

Automatic execution / cumulative PR budget:

{"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}

Projection check selection (not an executed result):

{"declared_in":"src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json","source_revision":"8288a004c333b0e338b3ad31fe823052a142b90a","status":"ready"}

Provider/settings: {"client":{"api_version":"2023-06-01","endpoint":"https://yibuapi.com/v1/messages","kind":"anthropic-messages-full-context-v1","max_output_tokens":4096,"max_read_bytes":32768,"max_request_bytes":262144,"max_requests":1,"max_tool_calls":12,"provider":"yibuapi"},"limits":{"token_stop_new_trials":200000},"model":{"effort":"not_applicable","name":"claude-opus-5-5"},"pricing":{"currency":"USD","expires_at":1793271228,"group":"default","max_input_tokens":1000000,"model":"claude-opus-5-5","rate_evidence":"https://github.com/InferMatrixLab/InferRSI/issues/215#issuecomment-6467531341","rates":{"cache_read":"0.4","cache_write":"10","input":"8","output":"40"},"request_fee":"0","unit":"per_million_tokens"},"spend_limit":{"amount":"200","currency":"USD"}}

Uncovered / unsupported paths (34)
  • uncovered: "README.md"
  • uncovered: "docs/knowledge-operations.md"
  • uncovered: "docs/operations.md"
  • uncovered: "docs/quickstart.md"
  • uncovered: "plugins/inferrsi/skills/inferrsi/SKILL.md"
  • uncovered: "plugins/inferrsi/skills/inferrsi/scripts/inferrsi.py"
  • uncovered: "plugins/workswarm/inferrsi/skills/inferrsi/SKILL.md"
  • uncovered: "plugins/workswarm/inferrsi/skills/inferrsi/scripts/inferrsi.py"
  • uncovered: "src/inferrsi/apps/cli/init.py"
  • uncovered: "src/inferrsi/apps/cli/arguments.py"
  • uncovered: "src/inferrsi/knowledge/_knowledge_errors.py"
  • uncovered: "src/inferrsi/knowledge/_skill_selection.py"
  • uncovered: "src/inferrsi/knowledge/data/query_aliases.json"
  • uncovered: "src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json"
  • uncovered: "src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-optimization-casebook.json"
  • uncovered: "src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/kernel-writing-guide.json"
  • uncovered: "src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/optimization-search-policy.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/methods/inference-optimization-casebook/cases/admission-drain-occupancy.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/methods/inference-optimization-casebook/cases/drama3-rejected-directions.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/methods/inference-optimization-casebook/cases/qwen-ascend-rejected-directions.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/methods/inference-optimization-casebook/cases/qwen-b300-rejected-directions.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/methods/kernel-writing-guide/cases/sglang-gdn-source-contract.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/methods/kernel-writing-guide/cases/vllm-gdn-source-contract.json"
  • uncovered: "src/inferrsi/knowledge/knowledge_find.py"
  • uncovered: "src/inferrsi/knowledge/knowledge_operations.py"
  • uncovered: "src/inferrsi/knowledge/skill_catalog.py"
  • uncovered: "tests/check_installed.py"
  • uncovered: "tests/fixtures/find/README.md"
  • uncovered: "tests/fixtures/find/labelled-queries.json"
  • uncovered: "tests/test_codex_plugin.py"
  • uncovered: "tests/test_knowledge_cli.py"
  • uncovered: "tests/test_knowledge_find.py"
  • uncovered: "tests/test_knowledge_install.py"
  • uncovered: "tools/skill_eval_snapshot.py"

@jiaran-king
jiaran-king marked this pull request as ready for review November 16, 2026 08:42
Agent-assisted-by: Codex (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by an agent on behalf of @Dong1017 (incremental review only; first review on this PR, no earlier human findings to repeat).

PR review verdict — DO NOT PASS

Reviewed base 016c7c7df023944f0f46d8e777d6d31f1df1764c (current main, also the merge base) and head 80ec573eb5b635a938cc6e215f1fd2dbed8c0d7f, 32 files, +2919/−62.

The find implementation itself looks careful: case discovery goes through the same KnowledgeReader.cases() that read uses, case resource checks are shared through case_resource_path, and every returned hit is opened through the reader before an open command is offered. The two blockers below are about admission and scope, not the ranking code.

Blockers

  1. Required knowledge-model-policy check fails at this head. The PR changes protected paths (src/inferrsi/knowledge/data/skill_library/** cases, knowledge_operations.py, skill_catalog.py, docs/knowledge-operations.md; see docs/knowledge-model-policy.md "exact protected scope"). The declarations run for 80ec573e reports "kind": "noncompliant" with "Material AI author 1: model is unapproved or suspended" and "Material AI author 3: model is unapproved or suspended" (run 43834106747). Authors 1 and 3 in the PR-body block declare model_id: "unknown" / effort: "unknown", and the policy says unknown values "stay truthful and do not qualify". Expected: either an exception approval from the approval owner per docs/knowledge-model-policy.md (it allows that "explicitly unknown settings may be considered"), or a revision whose protected-path content is authored only by qualifying declarations. Please do not change the declared values unless they are actually known.

  2. The change to the default agent entry point is not covered by the governing Issue's accepted scope. The PR says Refs #379. tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379's product decision currently reads: do not promote the task: 改进真实优化任务中的材料发现与使用 #380-tested guidance to the product entry and temporarily keep the existing catalogue and rg; it also says the tracker grants experiment: compare Skill Indexer and rg on MoE with GPT-6.1 Sol high #370/task: 支持每个优化迭代的知识库命中归因统计 #344 no extra implementation authority, and experiment: compare Skill Indexer and rg on MoE with GPT-6.1 Sol high #370 is the lane that compares retrieval entry points. This PR replaces the plugin's discovery guidance (plugins/inferrsi/skills/inferrsi/SKILL.md:124-150: knowledge research / knowledge search / collect --relevance are removed in favour of find → read) and makes find the README/quickstart default (README.md:227-228, docs/quickstart.md:55-56). The PR itself states that whether agents adopt the results and any optimisation benefit are not yet verified. Expected: a requester/maintainer note on tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 accepting find as the default entry, or a narrower revision that ships find / read --omit-resources as additional commands while leaving the plugin's default discovery guidance unchanged until that decision is made.

Optional (not blocking)

  • The quickstart example does not lead with the finding it is about. docs/quickstart.md:55 uses inferrsi find 'mask caching ran out of device memory'. Locally at this head (default kinds, and also with --kinds case --limit 1 as in the PR body), the Qwen Ascend case ranks first, but the finding that answers the query ("Cache per-step masks and prompt tensors → NPU OOM", rejected) is listed third, after two no_gain findings. caching/masks do not match cache/mask and nothing maps "out of … memory" to OOM, so within the case only mask (3 of 6 findings) and device (1 finding) score, and relevant_findings (src/inferrsi/knowledge/knowledge_find.py:218-238) puts the fusion-attention finding first. test_findings_are_ordered_by_query_overlap_and_truncated (tests/test_knowledge_find.py:136-143) asserts the right order only for the different wording cache per-step masks ran out of device memory. Either use that wording in the quickstart, or add an out of memory → oom synonym in query_aliases.json and pin the documented example in a test.
  • full-check (3.10) was still pending when this was written; full-check (3.11), installed-package (3.10/3.11) and pr-references were green.

Not verified: the ranking-quality table in the PR body (16/17, 21/34, 7/9, 9/9) and the 210,721 → 10,950 byte read-back reduction; I did not rerun the labelled-query fixtures against #380.

Checking scope: read the PR body, #379, the CI check list and the failed declarations log, docs/knowledge-model-policy.md, and the full diff of knowledge_find.py, cli.py, arguments.py, knowledge_operations.py, skill_catalog.py, _knowledge_errors.py, the plugin SKILL.md and README.md against 016c7c7d; skimmed the case JSON additions and query_aliases.json. Ran in a clean venv at the head (pip install -e '.[skills]'): pytest tests/test_knowledge_find.py tests/test_knowledge_cli.py tests/test_codex_plugin.py → 118 passed; ran the quickstart query and the printed open command (returns material and revision with resources omitted), and find --limit 0 (rejected with a clear error). Did not run the full suite, Ruff, the Windows/PowerShell path, the WorkSwarm/DSH plugin checks, or any agent-adoption evaluation.

Copy link
Copy Markdown
Collaborator

审阅了本 PR 的 base 016c7c7df023944f0f46d8e777d6d31f1df1764c 到当前 head 80ec573eb5b635a938cc6e215f1fd2dbed8c0d7f(完整 5 个提交)。当前 head 上不能 Approve,我和 @Dong1017 在评论 6911930950 中的两点阻塞结论一致,核对如下。

  1. 必选检查 knowledge-model-policy 在该 head 上为 noncompliant。declarations 日志写明:Material AI author 1: model is unapproved or suspended,Material AI author 3: model is unapproved or suspended(run 43834106747)。PR 正文声明里这两位的 model_id / effort 均为 unknown。按 docs/knowledge-model-policy.md,未知值应如实填写,但不能通过准入;需要已批准模型且 effort 为 xhigh 或 max,或由批准人按规范给出 exception。请不要为了过检而改写并不确实知道的声明值。该必选检查通过之前,这个 head 无法合入。

  2. 治理 issue tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 仍是 status:proposed,且当前产品决定是暂不把 task: 改进真实优化任务中的材料发现与使用 #380 所测短指引推广为产品入口,并临时保留现有目录和 rg;跟踪项也写明并不授予 experiment: compare Skill Indexer and rg on MoE with GPT-6.1 Sol high #370/task: 支持每个优化迭代的知识库命中归因统计 #344 额外实施权限。本 PR 却把插件默认发现路径改成了 find → read(plugins/inferrsi/skills/inferrsi/SKILL.md),并在 README / docs/quickstart.md 中把 find 写成默认入口。请先在 tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 上由请求者/维护者明确接受 find 作为默认入口,或把这次交付收窄为新增命令,先不改插件默认发现指引。

find 与 read/collect 的职责拆分、case_resource_path 共用、以及返回前用 reader 校验 hit,实现方向是清楚的。full-check / installed-package / pr-references 已通过。我没有额外的实现阻塞项,也没有逐条对照原始战役记录复核新加入的 findings,也没有重跑 PR 正文中的检索对照数字。请处理上述两点后再推新的 head。

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One compatibility issue with the documented discovery path.

@Dong1017 Dong1017 added area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice labels Nov 16, 2026
Use the existing case bodies to express the recorded decisions, limits, and follow-up checks. Keep the tests focused on which outcome is retrieved and the evidence it carries.

Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>

Copy link
Copy Markdown
Collaborator

审阅了自上次评论以来的增量:80ec573eb5b635a938cc6e215f1fd2dbed8c0d7f → 当前 head 791cf4d00d416357acc9a3095e2813a88e0f36a7(2 个提交:1431037fe138679a0b225e1501dbbc43f97a2150 重写 packaged findings,791cf4d00d416357acc9a3095e2813a88e0f36a7 刷新 skill projections)。当前 head 仍不能 Approve。

上次的两点阻塞都还在:

  1. 必选检查 knowledge-model-policy 在该 head 上仍是 noncompliant。最新 declarations(run 43841803683)报 Material AI author 1: model is unapproved or suspended。PR 正文已把 head_sha 更新到当前 head,并把作者 3(Codex)写成 gpt-6.1-sol / max,先前作者 3 的 unknown 问题不再出现;作者 1(Claude Code earlier search implementation)的 model_id / effort 仍是 unknown。按 docs/knowledge-model-policy.md,未知值应如实填写,但不能通过准入。需要已批准模型且 effort 为 xhigh 或 max,或由批准人按规范给出 exception。请不要为了过检而改写并不确实知道的声明值。

  2. tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 仍是 status:proposed,产品决定仍是暂不把短指引推广为产品入口。这次增量没有改插件 / README / quickstart 的发现路径,plugins/inferrsi/skills/inferrsi/SKILL.md、README.md 和 docs/quickstart.md 仍把 find 写成默认入口。请先在 tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 上由请求者/维护者明确接受,或把交付收窄为新增命令、先不改默认发现指引。

另有两点请在这个 head 上处理或更新:

  1. @david6666666 在当前 head 上的行内评论(4583888248)仍然成立:这次增量没有改 knowledge_find.py。技能 hit 仍使用 checkout: 身份,packaged-skill 投影仍被跳过;没有可选 PyYAML 时,命中会被标为不可读且不提供 open_argv。请按该评论补上 packaged collect / read 回退,或在插件指引里明确这条路径。

  2. 正文「实测结果」里,第三行仍写在 80ec573e 上复跑得到 7/9。这次 findings 重写会改变检索词面,可能使该数字以及未标明测量 head 的「重写后」数字失效。请在当前 head 上复跑并更新表,或标明哪些数字仍只对旧 head 有效。

full-check(3.10/3.11)、installed-package 和 pr-references 已通过。我没有逐条对照原始战役记录复核这次重写的 findings,也没有重跑检索对照。

Use the packaged projection and its revision to resolve a Skill that the base installation cannot open from find.

Agent-assisted-by: Codex (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head 866fd2aa2816652976c107d6ab09eb80d0343ef1.
Re-review after docs push that adds the packaged-skill PyYAML fallback guidance. Independent of author jiaran-king.

Blockers

  1. knowledge-model-policy still noncompliant on this head. declarations failed with Material AI author 1: model is unapproved or suspended (run 43846776491). PR-body author 1 still has model_id/effort = unknown. Keep unknown values truthful; obtain an approval-owner exception per docs/knowledge-model-policy.md, or revise protected-path authorship to qualifying declarations. Do not invent model ids.

  2. Default discovery promotion exceeds accepted #379 scope. #379 is still status:proposed and keeps the product decision: do not promote the #380 short guidance to the product entry; temporarily keep the existing catalogue and rg. This head still makes find → read the plugin/README/quickstart default (plugins/inferrsi/skills/inferrsi/SKILL.md, README.md, docs/quickstart.md). Expected: requester/maintainer acceptance on #379 for find as default, or narrow this PR to additive find/read --omit-resources without replacing the default discovery path.

  3. Branch is CONFLICTING vs current main. mergeable=false, mergeable_state=dirty. Conflicting paths include at least plugins/inferrsi/skills/inferrsi/scripts/inferrsi.py, tests/check_installed.py, and tests/test_knowledge_cli.py. Rebase/resolve onto current main before any Approve/merge.

Not blocking / observed

  • David's packaged fallback note (discussion_r4583888248) looks addressed in the plugin skill text on this head (collect --relevance → resource when PyYAML is missing). Keep that path through rebase.
  • Ranking implementation direction remains sound; ranking-quality table numbers were not re-run here.
  • No merge while dirty or while blockers 1–2 remain.

Checked / not checked

Checked: PR body, #379 accepted wording, declarations failure log, conflict paths via merge-tree, plugin SKILL fallback section, prior Dong/Amy/David findings. Did not re-run the labelled-query suites or full pytest on this conflicting head.

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One follow-up on the documented PyYAML fallback.

Copy link
Copy Markdown
Collaborator

审阅了自上次评论以来的增量:791cf4d00d416357acc9a3095e2813a88e0f36a7 → 当前 head 866fd2aa2816652976c107d6ab09eb80d0343ef1(1 个提交:docs: explain the packaged skill fallback without PyYAML,只改了 plugins/inferrsi/skills/inferrsi/SKILL.md 和 docs/operations.md)。当前 head 仍不能 Approve。

上次的两点阻塞都还在:

  1. 必选检查 knowledge-model-policy 在该 head 上仍是 noncompliant。最新 declarations(run 43846776491)报 Material AI author 1: model is unapproved or suspended。PR 正文已把 head_sha 更新到当前 head,但作者 1(Claude Code earlier search implementation)的 model_id / effort 仍是 unknown。按 docs/knowledge-model-policy.md,未知值应如实填写,但不能通过准入。需要已批准模型且 effort 为 xhigh 或 max,或由批准人按规范给出 exception。请不要为了过检而改写并不确实知道的声明值。

  2. tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 仍是 status:proposed。作者在 tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 评论 6913527304 中建议接受把 find 作为默认入口,但请求者/维护者尚未在 tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 记录决定,产品正文仍写“暂不将 task: 改进真实优化任务中的材料发现与使用 #380 所测短指引推广到产品入口”。这次增量只补充了无 PyYAML 时的回退说明,plugins/inferrsi/skills/inferrsi/SKILL.md、README.md 和 docs/quickstart.md 仍把 find 写成默认入口。请先在 tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 上得到明确接受,或把交付收窄为新增命令、先不改默认发现指引。

关于上次第 3 点:本次文档补充了 collect --relevance → resource 路径,但还不能算已解决。@david6666666 在当前 head 的行内评论(4584520972)指出,该回退按同一查询重新做 collect --relevance,不一定打开 find 选中的那条 skill;例子是 find 'cuda graph warmup' --kinds skill --limit 1 选中 kernel-writing-guide,而文档中的 collect --relevance 会选到 ascend-aclgraph-capture-replay。我没有在本机复跑这个例子,但文档把 collect --relevance "<same query>" 写成“查找该 packaged skill”,而该命令本身是一次新的相关性检索,确实不能保证对应同一条 hit。请按选中 hit 的路径去匹配 skill_documents.original,或把文档改写成一次新的搜索,而不是打开当前命中项。

上次第 4 点仍然存在:正文「实测结果」第三行仍写在 80ec573e 上复跑得到 7/9。本次文档增量没有更新该表,也未标明哪些数字仍只对旧 head 有效。请在当前 head 上复跑并更新,或明确标注测量 head。

当前 head 上 cursor[bot] 的 review(5918404550)为 CHANGES_REQUESTED / DO NOT PASS,其中知识模型策略与 #379 范围两点与上述 1、2 一致。本次增量是文档,不涉及新的精度或性能对比实验。我没有重跑检索对照,也没有在本机验证无 PyYAML 回退命令。

@jiangkuaixue123 jiangkuaixue123 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed target main, PR base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head 866fd2aa2816652976c107d6ab09eb80d0343ef1. Also checked integration against fetched main at 6c7e237935a22cc5e90e954d4f397376e753e7d8.

Delegation: the user directly requested an independent review of #466 and authorized posting review findings. Acting reviewer: Codex under @jiangkuaixue123 (repository write access verified); scope: code, tests and merge readiness; permitted action: publish this revision-bound Request changes review. This reviewer did not author or materially contribute to this PR.

当前不建议合并。以下是对已有意见的独立复核,不把重复问题计作新发现。

  1. [P2] 无 PyYAML 时的回退不能保证打开原来命中的 Skill。 已在当前 head、Python 3.11.15 禁用 site packages 的环境复现:find 'cuda graph warmup' --kinds skill --limit 1 --json 选中 kernel-writing-guide,路径为 methods/kernel-writing-guide/SKILL.md;而按 plugins/inferrsi/skills/inferrsi/SKILL.md:129-133 和 docs/operations.md:246-250 执行 collect --relevance 'cuda graph warmup' --limit 1 --material --json,返回的是 inferrsi.ascend.skills.ascend_aclgraph_capture_replay。两个命令都成功退出,但材料已经换了。请通过选中 hit 的身份/路径确定对应 package projection,并校验 skill_documents.original;或明确把回退描述为重新搜索、要求重新判断结果,不能称作读回该命中项。建议用这个例子补回归覆盖。确认 David 的现有意见 仍然成立。

  2. 必选 knowledge-model-policy 检查仍失败。 已读取当前 head 的最新 declarations 日志,明确报告 Material AI author 1: model is unapproved or suspended;也重新读取了 main 分支规则,确认该 context 仍是必选项。请保留真实的 unknown 声明,按仓库流程取得有权批准者的 exception,或提供符合规则的实际作者声明;普通代码 review 不能替代此项。

  3. 默认入口切换仍缺少记录下来的范围决定。 已读取 #379 正文及全部四条评论;最新提议 仍是在请求请求者/维护者接受 find 作为默认入口,未见接受记录。这不表示 #380 对旧短指引的负结果证明新工具无效;问题是本 PR 将 README、quickstart 和插件默认指引切到 find → read,而这项产品决定尚未记录。请在 #379 确认范围,或先保留原默认入口、把新命令作为增量能力交付。

  4. 当前分支存在合并冲突。 GitHub 返回 mergeable=false / mergeable_state=dirty;本地 git merge-tree --write-tree HEAD origin/main 在上述 fetched main 上也退出 1。当前实际冲突文件是 src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json。请结合最新 main 重新生成一致的索引/provenance,处理冲突后在新 head 上重跑检查并请求复审。此处的冲突路径是本次复核结果,不沿用较早评审列出的路径。

验证:

  • Python 3.11.15 / macOS,独立 detached worktree。运行 knowledge find/CLI/operations、skill catalog/index、Codex/WorkSwarm/DSH 插件及 module boundaries 共九个测试文件:217 passed, 2 skipped, 1 failed。
  • 唯一失败为 tests/test_dsh_plugin.py::test_remote_ssh_connection_and_scoped_operations,其 Node 检查返回 AssertionError ERR_ASSERTION。在单独的 base worktree 运行相同 tests/check_dsh_remote.cjs 也失败。因此不归因为本 PR 新增回归,但本次套件不是全绿。
  • ruff check src tests、git diff --check 通过;tools/build_skill_index.py --source-revision 1431037fe138679a0b225e1501dbbc43f97a2150 --check 通过(80 metadata records)。
  • 本地 editable 安装的 SCM 自动版本推导受已有非版本 Git 标签影响;仅测试安装使用 SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0,没有修改受审代码。上述结果不作为默认打包成功的证明。
  • 核对了新 findings 与仓库内原 Case 正文、检索评分、共享 reader/path 校验、精简读回和插件启动链;除上述已复现问题,未发现足够明确的新实现阻塞项。
  • 未运行全仓测试、Windows 实机、GPU/NPU 或真实 Agent 对照;未独立验证 PR 正文的优化收益/成本数据。本次是 PR 工程审查,不是 #379 的目标验收或 campaign PASS。

Move the find integration to the current Apps CLI and clarify that the no-PyYAML fallback searches packaged skills independently. Refresh the generated provenance in the next commit from this combined source revision.

Agent-assisted-by: Codex (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
…ression

Agent-assisted-by: Codex (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
@linyueqian

Copy link
Copy Markdown
Contributor

@david6666666 The 11-16 nightly lane (run 43891426266) failed at its last step, the label drift report: label drift: high priority: in the repository, missing from labels.json. Ruff, the full pytest step (4849 passed) and evidence validation passed in the same run.

high priority is on this PR only, added at 09:36Z, and the 11-15 nightly passed without it. The scheme in docs/agent-collaboration.md (Labels) wants group:value names, and its priority labels are priority:P0, P1 and P2 with a status label. Could the label come off this PR and out of the repository, or go into .github/labels.json under a group:value name first? The nightly stays red until one of the two happens.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base 891aadda6d4abc050def2baa2814b284f87808e5 (current merge-base / PR base after main merges) and head 8c0a7bfc39a9f95d610f6bd0b9f3682200ad0606. Re-review after 866fd2aa2816652976c107d6ab09eb80d0343ef1 → this head (merge main + regenerate provenance / install regression to 929). Independent of author jiaran-king.

Blockers

  1. knowledge-model-policy still noncompliant on this head. Required declarations failed with Material AI author 1: model is unapproved or suspended (run 43889327163). PR-body author 1 still has model_id/effort = unknown. Keep unknown values truthful; obtain an approval-owner exception per docs/knowledge-model-policy.md, or revise protected-path authorship to qualifying declarations. Do not invent model ids.

  2. Default discovery promotion still exceeds accepted #379 scope. #379 remains status:proposed and keeps the product decision not to promote the short guidance to the product entry (retain existing catalogue / rg). This head still makes find → read the plugin / README / quickstart default (plugins/inferrsi/skills/inferrsi/SKILL.md, README.md, docs/quickstart.md). Expected: requester/maintainer acceptance of #379 (or an explicit scoped exception recorded on the Issue) before shipping that promotion, or demote find to an opt-in / advanced path in those entry docs for this PR.

Prior peer finding (status)

jiangkuaixue123's PyYAML fallback mismatch (hit vs collect --relevance) is now documented as possibly selecting a different Skill (SKILL.md / docs/operations.md). That documentation addresses the “or state it clearly” branch of that CR; identity-bound open remains a desirable follow-up, not repeated here as a new steward blocker.

Checked

  • Diff vs base (~33 files / +2995/−67): knowledge_find, findings rewrite, CLI/plugin wiring, install count 929, provenance regen.
  • Required status: declarations → noncompliant on this exact head.
  • #379 labels still status:proposed.
  • Did not re-run the full pytest matrix locally; ready-lane green besides declarations is not merge eligibility while blockers above stand.

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by an authorized steward agent on behalf of @Dong1017 (steward triage; hosted Dong-bot on usage_limit).

Re-reviewed head 8c0a7bfc39a9f95d610f6bd0b9f3682200ad0606 → DO NOT PASS (formal Request changes): #466 (review)

Blockers unchanged in substance: knowledge-model-policy noncompliant (Material AI author 1 still unknown), and find default promotion still ahead of #379 acceptance.

@linyueqian — agreed the 11-16 nightly label-drift failure is from the ad-hoc high priority label on this PR (present in the repo, absent from .github/labels.json). Please remove that label from this PR (or open a separate labels.json PR if the team wants it catalogued). It is unrelated to the review blockers above.

Agent-assisted-by: Codex (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Agent-assisted-by: Codex (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One backend-ranking regression introduced on the current head.

Keep topic aliases in lexical matching and restore the indexed-directory lookup required by the Knowledge migration contract.

Agent-assisted-by: Codex (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
cursor[bot]
cursor Bot previously requested changes Nov 17, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base 891aadda6d4abc050def2baa2814b284f87808e5 and head c3364c25247259439a483e7fc9046173da64622b.
Independent re-review after prior Request changes on 8c0a7bfc39a9f95d610f6bd0b9f3682200ad0606 (author jiaran-king).

Prior blockers — cleared on this head

  1. knowledge-model-policy: required declarations / commit status knowledge-model-policy are success on c3364c25 (Material AI authors now carry approved model ids).
  2. #379 default-discovery scope: requester accepted find → read as the README / quickstart / plugin default (comment); maintainer acceptance recorded separately on #379 this round.

Remaining blocker

  1. WorkSwarm plugin still teaches the old discovery path. Accepted #379 scope requires the default in README, quickstart, and plugins. plugins/inferrsi/skills/inferrsi/SKILL.md leads with find, but plugins/workswarm/inferrsi/skills/inferrsi/SKILL.md (§ Research and diagnose, ~L42–49) still lists only knowledge research / knowledge search / collect --relevance then exact read. An agent following the installed WorkSwarm skill therefore misses cross-kind find, the displayed read command, and the documented PyYAML fallback. Update that section to lead with find "<problem or symptom>" + its displayed read args; keep the package-only commands as compatibility/fallback. (Same gap noted earlier as an incremental Dong1017 finding; still true on this head.)

Not repeated as new blockers

  • david6666666 backend-intent / 图模式 ranking note: head adds explicit-backend weighting + regressions (test_graph_mode_alias_matches_without_backend_preference, test_explicit_ascend_preference_survives_graph_mode_alias); Ready full-check green on this head.
  • Ranking-quality campaign numbers and live Agent comparisons were not re-run here.

Checked

  • PR body declaration vs declarations CI; #379 comments; WorkSwarm vs InferRSI plugin SKILL; Ready checks on c3364c25 (full-check / installed-package / host-compatibility / declarations success).

@Dong1017

Dong1017 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Steward note (Dong1017 delegate). Re-reviewed head c3364c25. DO NOT PASS — declarations/#379 product accept cleared; WorkSwarm plugin Research section still missing the accepted find → read default. Formal Request changes: #466 (review)

Maintainer accept of the #379 scope: #379 (comment)

Document the packaged Skill fallback and clarify no-match behavior in the Codex plugin.

Agent-assisted-by: Codex (author)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Comment thread docs/operations.md
@@ -35,6 +35,7 @@ Effect classes used below:
| Search packaged records | CLI `knowledge search` / `knowledge show`; catalog API | Read-only | Installed package catalog with the existing tier/repo/backend/scenario filters and IDs; these package-only filters exclude skills and cases from every other authority |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the old way we need to remove?

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent code review of head 33de913, without using other reviewers' comments or the PR discussion.

No actionable correctness defect found in the reviewed changes: lexical scoring and query aliases, authority-qualified open commands, case path validation, optional PyYAML handling, CLI resource omission, and both plugin launchers. Checked the new case metadata against the corresponding local source material; this is not independent reproduction of the historical hardware measurements.

Validation: ruff check src tests tools passed. The four affected suites (test_knowledge_find, test_knowledge_cli, test_knowledge_install, test_codex_plugin) cover 136 tests: 135 passed initially, and the remaining outside-checkout test passed after correcting my test environment to use an absolute PYTHONPATH. Independent probes opened 12 results from four English/Chinese queries and verified returned revisions and omitted resource inventories. Exact-head hosted ready checks are successful, including Python 3.10/3.11 full-check and installed-package, browser, Windows and macOS host jobs; knowledge-model-policy is also successful.

Review conclusion: no blocking finding from this review. Leaving COMMENT rather than approval: the local full suite was not completed. An initial broad run used the same unsuitable relative PYTHONPATH; a corrected run passed that earlier failing region before I stopped it. No merge or branch changes performed.

@jiangkuaixue123 jiangkuaixue123 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

按用户直接授权,对旧搜索是否仍进入实际优化流程做定向补充审查。Reviewed base 891aadd and head 33de913。已核对插件指引、两个 CLI 入口及 session → capture → collect 调用链,并以相同查询对比 capture callback 与 find 的结果;tests/test_campaign_task_selection.py 与 test_knowledge_delivery_is_retained_and_bound_to_round 共 4 项通过。下面是一条关于默认流程尚未统一的行内意见;这是定向补充反馈,未重新评估整个 PR 的合并条件。

- `read <ref> --revision <revision> --material` loads a selected exact revision.
Use the returned resource keys with `resource --help` when further material
is needed.
Use `find "<problem or symptom>"` to rank skills, imported records and casebook

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 默认优化流程仍会绕过 find,继续使用旧检索

这里把 find → read 写成默认发现入口,但同一份 Skill 的第 65–66、108–109、236–238 行仍要求普通优化走 campaign knowledge、题包优化走 benchmarks session knowledge;WorkSwarm 也有相同指引。沿实际调用链检查,这两个命令在 apps/cli/__init__.py 创建无 skill root 的 KnowledgeReader(),经 harness/optimization_session.py:210–214 调用 apps/campaign.py:49–56,最终仍执行 reader.collect(..., relevance_method="task-lexical-v1"),没有调用新 find。

在当前 head 用相同查询 cache per-step masks ran out of device memory、limit=3 复核:受管流程使用的 capture callback(backend=ascend)返回 training-input-pipeline、weight-loading-optimization、torchnpu-device-memory-lifecycle;直接 find 的第一条却是 case.qwen-ascend-rejected-directions。因此遵循优化章节的 Agent 仍会使用旧选择器,无法获得这里新增的直接 Case 排序能力;这不只是旧 CLI 命令为了兼容而保留。

请明确并统一这两个默认流程:如果优化任务也应默认使用新检索,应在受管入口接入它,同时保留冻结的 knowledge on/off 检查、正文/revision/context 留存及轮次关联,并增加经过实际 session/campaign 入口的回归测试;如果本 PR 有意只切换独立 research/diagnose,请在这里、quickstart 和 PR scope 明确该边界,避免读者以为实际优化运行也已切换。截图中的 knowledge search 可作为 package-only 兼容/过滤查询保留,无需仅因存在旧命令就删除它。

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One separate coverage gap between the new standalone retrieval path and campaign search attribution.

is needed.
Use `find "<problem or symptom>"` to rank skills, imported records and casebook
entries. Add `--kinds case` when you specifically need a past observation.
Open a promising hit using its displayed `open` command or the JSON

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also needs an attribution caveat for standalone find → read, separate from the managed optimization path. At this head I ran find RoPE --kinds case --json, executed its emitted open_argv successfully, then imported both tool events through campaign import-search in a synthetic Codex rollout. Neither event was recorded or flagged as unresolved; a partial import had no observations, and supplying --complete silently produced zero searches. Could we tell agents here not to mark these rounds complete unless the retrieval is independently captured? Alternatively, recognize the pair (or at least mark it unresolved) and add an importer fixture. A returned hit, an actual read, and adoption should remain separate.

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 33de913c. All seven comments from my earlier pass are addressed, each with a regression test that would have caught the original:

  • unverified() now reads the structured field instead of substring-matching the flattened dict (test_find_preserves_a_structured_verification_limitation).
  • Repeated query terms contribute once — "vae tiling vae tiling" now scores exactly the same as "vae tiling" (test_repeated_query_terms_contribute_once).
  • Installed skills carry source_location: installed and print an installed label, so the hit no longer reads as a user checkout.
  • Skill descriptions go through YAML when it is available, with the regex kept only as the base-install fallback; inference-optimization-casebook no longer returns TensorRT-LLM''s.
  • Record body excludes title/summary/tags/applicability, which are scored separately (test_record_body_excludes_fields_scored_separately).
  • --kinds record no longer resolves a skill root it does not use (test_record_only_search_ignores_an_unused_skill_root).
  • The case fixture writes bytes, so the Windows CRLF failure is gone.

The backend-preference work in c3364c25 reads well too: named_backends is factored out of _skill_selection so find and collect share one definition, and the non-matching backend gets a 0.5 multiplier rather than being dropped, which keeps the Ascend/H20 H3 records visible with their applies to intact.

Checked on 33de913c: tests/test_knowledge_find.py 60/60 pass locally; CI full-check (3.10), full-check (3.11), declarations, knowledge-model-policy and host-compatibility (windows-latest) all pass; mergeable: MERGEABLE.

One non-blocking note left inline. Approving — as a reviewer I cannot clear the bot's outstanding request-changes on c3364c25, so that still needs its own re-run.

return set(re.findall(r"\w+", cue.casefold()))


def named_backends(terms: set[str]) -> set[str]:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

named_backends only recognises vendor words, so "H200 minimax h3 parallel" yields no backend and audit-h3-runtime-residency (Apple Metal) still ranks above the CUDA records; adding nvidia or cuda fixes it. Mapping the common GPU SKUs (h200/b200/h20 -> cuda) would make the discrimination work for the way people actually phrase these queries. Non-blocking.

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@david6666666
david6666666 enabled auto-merge (squash) November 17, 2026 07:12
@david6666666
david6666666 merged commit 61303fa into InferMatrixLab:main Nov 17, 2026
14 checks passed
BeatSeat added a commit to BeatSeat/InferRSI that referenced this pull request Nov 17, 2026
main installs 929 packaged resources (InferMatrixLab#466); the measurement-validity
reference adds one, so the installed count is 930, verified on a wheel built
from a clean archive of this head.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants