Repository navigation
feat(knowledge): unify search across cases, skills, and records - #466
Conversation
…PyYAML Each casebook entry already recorded the directions tried, their outcome and when a retry would be justified, but only as prose or a Markdown table that ranking could not weigh. Add a `findings` list to all seven cases: direction, outcome, recorded reason, revisit condition, retrieval keywords and `scope`. Every text field is taken from the existing case body; no claim, cause or measurement is added and no existing field changes. `scope` states what would have to change for an observation to stop holding: `local` (that host or harness), `regime_bound` (the cohort the case names) or `general` (a property of the method). It is not derivable from the outcome -- a rejection can be bound to one resolution or hold everywhere. Of 35 findings, 24 are regime_bound, 6 general and 5 local. The source-contract cases are regime_bound: each reads one pinned upstream commit and says so in its own applicability, and a later revision can change the dispatch it describes. Reading a case parses no skill frontmatter, but importing the skill catalog required PyYAML, so a base install could not discover or open one. The parser is now imported where frontmatter is parsed. Its absence raises MissingSkillsExtra, a ValueError subclass in a module of its own so catching it never imports the parser, and `collect` lets it through rather than reporting "nothing found" for an install problem. The snapshot tool relied on the old import-time failure to stop before side effects, and now asks for the parser explicitly at that point. The reader exposes `cases()` and `case_resource_path()` so a caller ranking cases resolves them by the same rule `read` uses. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=max] (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
…ndings Mechanical follow-up. The generator pins each card's source hashes to a revision, so the three skills that own the edited cases -- the optimization casebook, the kernel-writing guide and the search policy -- held hashes for the case files as they were before findings were added, and `indexed_skill` rejected those projections as stale. Regenerated with tools/build_skill_index.py against the previous commit. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=max] (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Port the verified find and read flow to the current Knowledge package and Apps-facing CLI. Keep query fixtures, compact exact reads, and standalone plugin coverage together. Signed-off-by: Zhou ziheng <jiaranran2@gmail.com> Agent-assisted-by: Claude Code (author) Agent-assisted-by: Codex (author)
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision 8288a004c333b0e338b3ad31fe823052a142b90a --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
python -m json.tool optimization-search-policy/plan/tasks.json
python -m json.tool optimization-search-policy/plan/schema.jsonUncovered scope by location: Canonical library paths: 6; Generated projection paths: 4; Other paths: 24. Path counts are not claim counts; no metadata-only conclusion is inferred. Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Frozen exam hashes: Selection/task metadata (not a relevance verdict): {"adapter":"kv-admission-diagnostic-v1","basis":"trusted_path_mapping","matched_paths":["src/inferrsi/knowledge/data/skill_library/methods/optimization-search-policy/cases/sarathi-chunk-budget.json"],"required_answer_fields":["diagnosis","evidence","next_check","reuse_authorized"],"tasks":[{"id":"case-05","provenance":"Synthetic finite-state observations; not a production trace or engine benchmark.","request":"Diagnose this blocked request at the recorded decision boundary; propose the next read-only check.","split":"holdout"}]}Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json","source_revision":"8288a004c333b0e338b3ad31fe823052a142b90a","status":"ready"}Provider/settings: Uncovered / unsupported paths (34)
|
Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
|
Posted by an agent on behalf of @Dong1017 (incremental review only; first review on this PR, no earlier human findings to repeat). PR review verdict — DO NOT PASS Reviewed base The Blockers
Optional (not blocking)
Not verified: the ranking-quality table in the PR body (16/17, 21/34, 7/9, 9/9) and the 210,721 → 10,950 byte read-back reduction; I did not rerun the labelled-query fixtures against #380. Checking scope: read the PR body, #379, the CI check list and the failed |
|
审阅了本 PR 的 base
|
david6666666
left a comment
There was a problem hiding this comment.
One compatibility issue with the documented discovery path.
Use the existing case bodies to express the recorded decisions, limits, and follow-up checks. Keep the tests focused on which outcome is retrieved and the evidence it carries. Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
|
审阅了自上次评论以来的增量: 上次的两点阻塞都还在:
另有两点请在这个 head 上处理或更新:
|
Use the packaged projection and its revision to resolve a Skill that the base installation cannot open from find. Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head 866fd2aa2816652976c107d6ab09eb80d0343ef1.
Re-review after docs push that adds the packaged-skill PyYAML fallback guidance. Independent of author jiaran-king.
Blockers
-
knowledge-model-policystill noncompliant on this head.declarationsfailed withMaterial AI author 1: model is unapproved or suspended(run 43846776491). PR-body author 1 still hasmodel_id/effort=unknown. Keep unknown values truthful; obtain an approval-owner exception perdocs/knowledge-model-policy.md, or revise protected-path authorship to qualifying declarations. Do not invent model ids. -
Default discovery promotion exceeds accepted #379 scope. #379 is still
status:proposedand keeps the product decision: do not promote the #380 short guidance to the product entry; temporarily keep the existing catalogue andrg. This head still makesfind → readthe plugin/README/quickstart default (plugins/inferrsi/skills/inferrsi/SKILL.md,README.md,docs/quickstart.md). Expected: requester/maintainer acceptance on #379 forfindas default, or narrow this PR to additivefind/read --omit-resourceswithout replacing the default discovery path. -
Branch is CONFLICTING vs current
main.mergeable=false,mergeable_state=dirty. Conflicting paths include at leastplugins/inferrsi/skills/inferrsi/scripts/inferrsi.py,tests/check_installed.py, andtests/test_knowledge_cli.py. Rebase/resolve onto currentmainbefore any Approve/merge.
Not blocking / observed
- David's packaged fallback note (discussion_r4583888248) looks addressed in the plugin skill text on this head (
collect --relevance→resourcewhen PyYAML is missing). Keep that path through rebase. - Ranking implementation direction remains sound; ranking-quality table numbers were not re-run here.
- No merge while dirty or while blockers 1–2 remain.
Checked / not checked
Checked: PR body, #379 accepted wording, declarations failure log, conflict paths via merge-tree, plugin SKILL fallback section, prior Dong/Amy/David findings. Did not re-run the labelled-query suites or full pytest on this conflicting head.
david6666666
left a comment
There was a problem hiding this comment.
One follow-up on the documented PyYAML fallback.
|
审阅了自上次评论以来的增量: 上次的两点阻塞都还在:
关于上次第 3 点:本次文档补充了 上次第 4 点仍然存在:正文「实测结果」第三行仍写在 当前 head 上 cursor[bot] 的 review(5918404550)为 CHANGES_REQUESTED / DO NOT PASS,其中知识模型策略与 #379 范围两点与上述 1、2 一致。本次增量是文档,不涉及新的精度或性能对比实验。我没有重跑检索对照,也没有在本机验证无 PyYAML 回退命令。 |
jiangkuaixue123
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed target main, PR base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head 866fd2aa2816652976c107d6ab09eb80d0343ef1. Also checked integration against fetched main at 6c7e237935a22cc5e90e954d4f397376e753e7d8.
Delegation: the user directly requested an independent review of #466 and authorized posting review findings. Acting reviewer: Codex under @jiangkuaixue123 (repository write access verified); scope: code, tests and merge readiness; permitted action: publish this revision-bound Request changes review. This reviewer did not author or materially contribute to this PR.
当前不建议合并。以下是对已有意见的独立复核,不把重复问题计作新发现。
-
[P2] 无 PyYAML 时的回退不能保证打开原来命中的 Skill。 已在当前 head、Python 3.11.15 禁用 site packages 的环境复现:
find 'cuda graph warmup' --kinds skill --limit 1 --json选中kernel-writing-guide,路径为methods/kernel-writing-guide/SKILL.md;而按plugins/inferrsi/skills/inferrsi/SKILL.md:129-133和docs/operations.md:246-250执行collect --relevance 'cuda graph warmup' --limit 1 --material --json,返回的是inferrsi.ascend.skills.ascend_aclgraph_capture_replay。两个命令都成功退出,但材料已经换了。请通过选中 hit 的身份/路径确定对应 package projection,并校验skill_documents.original;或明确把回退描述为重新搜索、要求重新判断结果,不能称作读回该命中项。建议用这个例子补回归覆盖。确认 David 的现有意见 仍然成立。 -
必选
knowledge-model-policy检查仍失败。 已读取当前 head 的最新 declarations 日志,明确报告Material AI author 1: model is unapproved or suspended;也重新读取了 main 分支规则,确认该 context 仍是必选项。请保留真实的 unknown 声明,按仓库流程取得有权批准者的 exception,或提供符合规则的实际作者声明;普通代码 review 不能替代此项。 -
默认入口切换仍缺少记录下来的范围决定。 已读取 #379 正文及全部四条评论;最新提议 仍是在请求请求者/维护者接受
find作为默认入口,未见接受记录。这不表示 #380 对旧短指引的负结果证明新工具无效;问题是本 PR 将 README、quickstart 和插件默认指引切到find → read,而这项产品决定尚未记录。请在 #379 确认范围,或先保留原默认入口、把新命令作为增量能力交付。 -
当前分支存在合并冲突。 GitHub 返回
mergeable=false / mergeable_state=dirty;本地git merge-tree --write-tree HEAD origin/main在上述 fetched main 上也退出 1。当前实际冲突文件是src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json。请结合最新 main 重新生成一致的索引/provenance,处理冲突后在新 head 上重跑检查并请求复审。此处的冲突路径是本次复核结果,不沿用较早评审列出的路径。
验证:
- Python 3.11.15 / macOS,独立 detached worktree。运行 knowledge find/CLI/operations、skill catalog/index、Codex/WorkSwarm/DSH 插件及 module boundaries 共九个测试文件:217 passed, 2 skipped, 1 failed。
- 唯一失败为
tests/test_dsh_plugin.py::test_remote_ssh_connection_and_scoped_operations,其 Node 检查返回AssertionError ERR_ASSERTION。在单独的 base worktree 运行相同tests/check_dsh_remote.cjs也失败。因此不归因为本 PR 新增回归,但本次套件不是全绿。 ruff check src tests、git diff --check通过;tools/build_skill_index.py --source-revision 1431037fe138679a0b225e1501dbbc43f97a2150 --check通过(80 metadata records)。- 本地 editable 安装的 SCM 自动版本推导受已有非版本 Git 标签影响;仅测试安装使用
SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0,没有修改受审代码。上述结果不作为默认打包成功的证明。 - 核对了新 findings 与仓库内原 Case 正文、检索评分、共享 reader/path 校验、精简读回和插件启动链;除上述已复现问题,未发现足够明确的新实现阻塞项。
- 未运行全仓测试、Windows 实机、GPU/NPU 或真实 Agent 对照;未独立验证 PR 正文的优化收益/成本数据。本次是 PR 工程审查,不是 #379 的目标验收或 campaign PASS。
Move the find integration to the current Apps CLI and clarify that the no-PyYAML fallback searches packaged skills independently. Refresh the generated provenance in the next commit from this combined source revision. Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
…ression Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
|
@david6666666 The 11-16 nightly lane (run 43891426266) failed at its last step, the label drift report:
|
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 891aadda6d4abc050def2baa2814b284f87808e5 (current merge-base / PR base after main merges) and head 8c0a7bfc39a9f95d610f6bd0b9f3682200ad0606. Re-review after 866fd2aa2816652976c107d6ab09eb80d0343ef1 → this head (merge main + regenerate provenance / install regression to 929). Independent of author jiaran-king.
Blockers
-
knowledge-model-policystill noncompliant on this head. Requireddeclarationsfailed withMaterial AI author 1: model is unapproved or suspended(run 43889327163). PR-body author 1 still hasmodel_id/effort=unknown. Keep unknown values truthful; obtain an approval-owner exception perdocs/knowledge-model-policy.md, or revise protected-path authorship to qualifying declarations. Do not invent model ids. -
Default discovery promotion still exceeds accepted #379 scope. #379 remains
status:proposedand keeps the product decision not to promote the short guidance to the product entry (retain existing catalogue /rg). This head still makesfind → readthe plugin / README / quickstart default (plugins/inferrsi/skills/inferrsi/SKILL.md,README.md,docs/quickstart.md). Expected: requester/maintainer acceptance of #379 (or an explicit scoped exception recorded on the Issue) before shipping that promotion, or demotefindto an opt-in / advanced path in those entry docs for this PR.
Prior peer finding (status)
jiangkuaixue123's PyYAML fallback mismatch (hit vs collect --relevance) is now documented as possibly selecting a different Skill (SKILL.md / docs/operations.md). That documentation addresses the “or state it clearly” branch of that CR; identity-bound open remains a desirable follow-up, not repeated here as a new steward blocker.
Checked
- Diff vs base (~33 files / +2995/−67):
knowledge_find, findings rewrite, CLI/plugin wiring, install count 929, provenance regen. - Required status:
declarations→ noncompliant on this exact head. - #379 labels still
status:proposed. - Did not re-run the full pytest matrix locally; ready-lane green besides declarations is not merge eligibility while blockers above stand.
|
Posted by an authorized steward agent on behalf of @Dong1017 (steward triage; hosted Dong-bot on Re-reviewed head Blockers unchanged in substance: @linyueqian — agreed the 11-16 nightly label-drift failure is from the ad-hoc |
Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
david6666666
left a comment
There was a problem hiding this comment.
One backend-ranking regression introduced on the current head.
Keep topic aliases in lexical matching and restore the indexed-directory lookup required by the Knowledge migration contract. Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 891aadda6d4abc050def2baa2814b284f87808e5 and head c3364c25247259439a483e7fc9046173da64622b.
Independent re-review after prior Request changes on 8c0a7bfc39a9f95d610f6bd0b9f3682200ad0606 (author jiaran-king).
Prior blockers — cleared on this head
knowledge-model-policy: requireddeclarations/ commit statusknowledge-model-policyare success onc3364c25(Material AI authors now carry approved model ids).- #379 default-discovery scope: requester accepted
find → readas the README / quickstart / plugin default (comment); maintainer acceptance recorded separately on #379 this round.
Remaining blocker
- WorkSwarm plugin still teaches the old discovery path. Accepted #379 scope requires the default in README, quickstart, and plugins.
plugins/inferrsi/skills/inferrsi/SKILL.mdleads withfind, butplugins/workswarm/inferrsi/skills/inferrsi/SKILL.md(§ Research and diagnose, ~L42–49) still lists onlyknowledge research/knowledge search/collect --relevancethen exactread. An agent following the installed WorkSwarm skill therefore misses cross-kindfind, the displayed read command, and the documented PyYAML fallback. Update that section to lead withfind "<problem or symptom>"+ its displayed read args; keep the package-only commands as compatibility/fallback. (Same gap noted earlier as an incremental Dong1017 finding; still true on this head.)
Not repeated as new blockers
- david6666666 backend-intent /
图模式ranking note: head adds explicit-backend weighting + regressions (test_graph_mode_alias_matches_without_backend_preference,test_explicit_ascend_preference_survives_graph_mode_alias); Ready full-check green on this head. - Ranking-quality campaign numbers and live Agent comparisons were not re-run here.
Checked
- PR body declaration vs declarations CI; #379 comments; WorkSwarm vs InferRSI plugin SKILL; Ready checks on
c3364c25(full-check / installed-package / host-compatibility / declarations success).
|
Steward note (Dong1017 delegate). Re-reviewed head Maintainer accept of the #379 scope: #379 (comment) |
Document the packaged Skill fallback and clarify no-match behavior in the Codex plugin. Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
| @@ -35,6 +35,7 @@ Effect classes used below: | |||
| | Search packaged records | CLI `knowledge search` / `knowledge show`; catalog API | Read-only | Installed package catalog with the existing tier/repo/backend/scenario filters and IDs; these package-only filters exclude skills and cases from every other authority | | |||
There was a problem hiding this comment.
Is the old way we need to remove?
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent code review of head 33de913, without using other reviewers' comments or the PR discussion.
No actionable correctness defect found in the reviewed changes: lexical scoring and query aliases, authority-qualified open commands, case path validation, optional PyYAML handling, CLI resource omission, and both plugin launchers. Checked the new case metadata against the corresponding local source material; this is not independent reproduction of the historical hardware measurements.
Validation: ruff check src tests tools passed. The four affected suites (test_knowledge_find, test_knowledge_cli, test_knowledge_install, test_codex_plugin) cover 136 tests: 135 passed initially, and the remaining outside-checkout test passed after correcting my test environment to use an absolute PYTHONPATH. Independent probes opened 12 results from four English/Chinese queries and verified returned revisions and omitted resource inventories. Exact-head hosted ready checks are successful, including Python 3.10/3.11 full-check and installed-package, browser, Windows and macOS host jobs; knowledge-model-policy is also successful.
Review conclusion: no blocking finding from this review. Leaving COMMENT rather than approval: the local full suite was not completed. An initial broad run used the same unsuitable relative PYTHONPATH; a corrected run passed that earlier failing region before I stopped it. No merge or branch changes performed.
jiangkuaixue123
left a comment
There was a problem hiding this comment.
按用户直接授权,对旧搜索是否仍进入实际优化流程做定向补充审查。Reviewed base 891aadd and head 33de913。已核对插件指引、两个 CLI 入口及 session → capture → collect 调用链,并以相同查询对比 capture callback 与 find 的结果;tests/test_campaign_task_selection.py 与 test_knowledge_delivery_is_retained_and_bound_to_round 共 4 项通过。下面是一条关于默认流程尚未统一的行内意见;这是定向补充反馈,未重新评估整个 PR 的合并条件。
| - `read <ref> --revision <revision> --material` loads a selected exact revision. | ||
| Use the returned resource keys with `resource --help` when further material | ||
| is needed. | ||
| Use `find "<problem or symptom>"` to rank skills, imported records and casebook |
There was a problem hiding this comment.
[P2] 默认优化流程仍会绕过 find,继续使用旧检索
这里把 find → read 写成默认发现入口,但同一份 Skill 的第 65–66、108–109、236–238 行仍要求普通优化走 campaign knowledge、题包优化走 benchmarks session knowledge;WorkSwarm 也有相同指引。沿实际调用链检查,这两个命令在 apps/cli/__init__.py 创建无 skill root 的 KnowledgeReader(),经 harness/optimization_session.py:210–214 调用 apps/campaign.py:49–56,最终仍执行 reader.collect(..., relevance_method="task-lexical-v1"),没有调用新 find。
在当前 head 用相同查询 cache per-step masks ran out of device memory、limit=3 复核:受管流程使用的 capture callback(backend=ascend)返回 training-input-pipeline、weight-loading-optimization、torchnpu-device-memory-lifecycle;直接 find 的第一条却是 case.qwen-ascend-rejected-directions。因此遵循优化章节的 Agent 仍会使用旧选择器,无法获得这里新增的直接 Case 排序能力;这不只是旧 CLI 命令为了兼容而保留。
请明确并统一这两个默认流程:如果优化任务也应默认使用新检索,应在受管入口接入它,同时保留冻结的 knowledge on/off 检查、正文/revision/context 留存及轮次关联,并增加经过实际 session/campaign 入口的回归测试;如果本 PR 有意只切换独立 research/diagnose,请在这里、quickstart 和 PR scope 明确该边界,避免读者以为实际优化运行也已切换。截图中的 knowledge search 可作为 package-only 兼容/过滤查询保留,无需仅因存在旧命令就删除它。
david6666666
left a comment
There was a problem hiding this comment.
One separate coverage gap between the new standalone retrieval path and campaign search attribution.
| is needed. | ||
| Use `find "<problem or symptom>"` to rank skills, imported records and casebook | ||
| entries. Add `--kinds case` when you specifically need a past observation. | ||
| Open a promising hit using its displayed `open` command or the JSON |
There was a problem hiding this comment.
This also needs an attribution caveat for standalone find → read, separate from the managed optimization path. At this head I ran find RoPE --kinds case --json, executed its emitted open_argv successfully, then imported both tool events through campaign import-search in a synthetic Codex rollout. Neither event was recorded or flagged as unresolved; a partial import had no observations, and supplying --complete silently produced zero searches. Could we tell agents here not to mark these rounds complete unless the retrieval is independently captured? Alternatively, recognize the pair (or at least mark it unresolved) and add an importer fixture. A returned hit, an actual read, and adoption should remain separate.
david6666666
left a comment
There was a problem hiding this comment.
Re-reviewed at 33de913c. All seven comments from my earlier pass are addressed, each with a regression test that would have caught the original:
unverified()now reads the structured field instead of substring-matching the flattened dict (test_find_preserves_a_structured_verification_limitation).- Repeated query terms contribute once —
"vae tiling vae tiling"now scores exactly the same as"vae tiling"(test_repeated_query_terms_contribute_once). - Installed skills carry
source_location: installedand print aninstalledlabel, so the hit no longer reads as a user checkout. - Skill descriptions go through YAML when it is available, with the regex kept only as the base-install fallback;
inference-optimization-casebookno longer returnsTensorRT-LLM''s. - Record
bodyexcludestitle/summary/tags/applicability, which are scored separately (test_record_body_excludes_fields_scored_separately). --kinds recordno longer resolves a skill root it does not use (test_record_only_search_ignores_an_unused_skill_root).- The case fixture writes bytes, so the Windows CRLF failure is gone.
The backend-preference work in c3364c25 reads well too: named_backends is factored out of _skill_selection so find and collect share one definition, and the non-matching backend gets a 0.5 multiplier rather than being dropped, which keeps the Ascend/H20 H3 records visible with their applies to intact.
Checked on 33de913c: tests/test_knowledge_find.py 60/60 pass locally; CI full-check (3.10), full-check (3.11), declarations, knowledge-model-policy and host-compatibility (windows-latest) all pass; mergeable: MERGEABLE.
One non-blocking note left inline. Approving — as a reviewer I cannot clear the bot's outstanding request-changes on c3364c25, so that still needs its own re-run.
| return set(re.findall(r"\w+", cue.casefold())) | ||
|
|
||
|
|
||
| def named_backends(terms: set[str]) -> set[str]: |
There was a problem hiding this comment.
named_backends only recognises vendor words, so "H200 minimax h3 parallel" yields no backend and audit-h3-runtime-residency (Apple Metal) still ranks above the CUDA records; adding nvidia or cuda fixes it. Mapping the common GPU SKUs (h200/b200/h20 -> cuda) would make the discrimination work for the way people actually phrase these queries. Non-blocking.
main installs 929 packaged resources (InferMatrixLab#466); the measurement-validity reference adds one, so the installed count is 930, verified on a wheel built from a clean archive of this head. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
Scope
Refs #379
Agents can describe an optimization problem, find relevant guides, records, and historical cases in one query, and open the source text with its provenance, applicability, limitations, and revision. This PR delivers that local search and read workflow.
Previous search behavior
knowledge search/knowledge researchcollect --relevancecollect/readIn earlier replays, some relevant cases fell outside the selected results. Some selected material was cut off in the outer tool response. Finding a match and delivering its text are separate steps.
Changes
inferrsi find "<problem description>"uses a shared term-based ranking for registered cases, skills, and packaged records. It weights titles, summaries, tags, and findings more heavily, precomputes term frequencies, and gives long bodies less weight. It supports word boundaries, common separators, Chinese and English aliases, and the--kindsfilter.findingsfrom the original text of seven cases so searches can match the direction tried, outcome, reasoning, and conditions for revisiting it. Each preview shows the applicability or bounds recorded in the source.opencommand, withopen_argvin JSON. The search uses the same reader to check that a hit can be opened.read --material --omit-resourcesreturns the text, sources, and revision. A full read includes the resource inventory.find → readas the default discovery workflow in the README, operations guide, and plugin instructions. The existing exact lookup, read, and resource commands remain available to callers. The Codex and WorkSwarm plugins read material through their respective installed launchers.This query returns the Qwen Ascend case and its open command. The read response includes the revision for later citations pinned to that version.
Review follow-up
source_location. An installed Skill or Case opens its original body through thecheckout:reader route with the installed root.--kinds recordskips the unused skill root.not,without,before, andafterstill contribute. When the original query explicitly names one supported backend, results registered for a different backend receive half their lexical score.图模式can matchcuda graph, and the explicit昇腾in昇腾 图模式gives Ascend results priority over otherwise equal CUDA results. Both cases have regression coverage. David's review_indexed_directoriesinterface required by the Knowledge migration contract. The existing reader andfindshare the validated index loader.find → readin the WorkSwarm entry skill and document the packaged Skill fallback. The Codex entry skill now describes whenfindreturnsno_match. Steward re-reviewSearch replay results
The kind-labeled sets check whether the expected kind appears in the top three.
8c0a7bfc51aad393ce5c8865On
51aad393, the evidence-based labels registered three fewer Case-kind hits and one more Skill-kind hit than on8c0a7bfc. Temporarily restoring repeated-term scoring on51aad393recovered all three Case-kind hits in a local diagnostic. The latest revision gained one Record-kind hit and lost one Case-kind hit. The lost hit wascase.sarathi-chunk-budgeton a Ref2VA delivery-latency query. The frozen queries check the exact target ID and full-text read-back.Both
CUDA graph captureandHow should I warm up CUDA graph capture?now rankpytorch.cuda.compiler.cudagraph_shapes_and_lifetimesfirst. The unrelatedskill-evolution-playbookdrops out of the top three for the full-sentence query. The query about caching per-step masks rankscase.qwen-ascend-rejected-directionsfirst. The question about word error rate under concurrent service rankscase.drama3-rejected-directionsfirst.The frozen set includes five Qwen queries and four about shared-resource hashing. The returned text matched the frozen source byte for byte and included the same provenance, applicability, and limitations. Reopening at the returned revision gave the same result. The follow-up
c3364c25reproduced the figures in thece5c8865column.On
8c0a7bfc, the JSON response forpackage:inferrsi.common.skills.optimization_search_policywent from 223,782 bytes for a full read to 23,757 bytes for a compact read. The text, provenance, applicability, and revision matched.Qwen-Image retrieval comparison
This section records the earlier agent-run comparison. The current review revision was evaluated with the fixed-query replays above.
Two fresh GPT-6 Luna / max subagent sessions with independent contexts ran the scaled
qwen21-denoise-nputask package from #338. They ran in order A then B on the same Ascend 910B2C, comparing the previoussession knowledgeentry point withfind → read. Both sessions had access to the knowledge base and could use historical material returned by their entry point. The corpus, delivery method, and agent-written queries could differ. Each session completed eight distinct candidates and an independent final run withpairs=2, samples=3, warmups=1. The benchmark used five correctness cases, three of which were timed.find → readReturned items are counted across queries, including repeat hits. Full-text reads and uses count distinct items. A received full text three times across two items. B opened full text six times across four items. Use is counted where the agent recorded a link from the material to a candidate or a decision about which direction to try. Cached input is included in total input. The campaign token estimate follows #338 and uses cumulative host counts sampled before the start and near final completion. Uncached input plus output and the campaign estimate are usage figures. Billing requires separate accounting.
B recorded more uses of retrieved material and had lower campaign wall time and total session tokens. A's final candidate had lower median latency in all three timed cases. Each retrieval method ran in one agent session, with three queries per session. This comparison was selected after observing the results, and its conclusions apply to these two sessions.
Final results by case, round-by-round results, and retrieved material
Each final run paired the baseline and candidate measurements within the same arm for every case.
Cumulative tokens are estimated from host counts sampled near the end of each round, minus the count before the start. These samples give approximate usage by round. The campaign retained the other candidates and their feedback.
Full-text material delivered or read during the sessions
package:inferrsi.common.skills.skill_evolution_playbooksha256:fdba8ee31bc48d8a3cec0d5da3200324432b2b1782c3199c1de163042ec586d5package:inferrsi.ascend.skills.ascend_operator_tuningsha256:54f3556be2fbd4b5e6e2af68c7b4df9a7c1d2c53fde23150132063d20a25f267vllm_omni.ascend.diffusion.qwen_image_21_eager_t2isha256:b1740d8dd28f3c2f7ec2e4fbb9233a7ce311f52210fa3c6365bb320ad191e142case.qwen-ascend-rejected-directionssha256:039a074c881a924327549416d143fcf7cb550493e87352113b45a08d8717827acase.qwen-b300-rejected-directionssha256:0a9a9890243d4fd8a58989fbf7e27cab65cacc60f2471892da00ffc3b19e49d6torch-npusha256:dab34d1dcd1fe61b1fd1223d592a3d1ad2d878ae5d3f6d4edf8d1ec29487462fThe historical entries were available as leads for the experiments. The counts reflect what the sessions attributed to those entries. Causal effects require separate evidence.
The runs used
pairs=2, samples=3, warmups=1for every candidate and final, covering five correctness cases and timing three. The exact wait for the initial device allocation is unknown. Each session kept the same device across rounds after allocation. Campaign wall time includes agent reasoning, retrieval, editing, and evaluation. The benchmark scope is scaled synthetic denoising. Real-weight image quality and end-to-end service performance would require separate evidence. The original artifacts and full evidence tree are held privately.Checks and evidence limits
33de913c, 43 plugin and runtime checks passed. In an extracted WorkSwarm archive with PyYAML absent, defaultfindreturned the Qwen case and itsopen_argvread the body, sources, and revision. The documented fallback fetched a packaged Skill whose body matched the source byte for byte. An unmatched query returnedno_match.c3364c25, 263 affected tests passed, including the compatibility suite and both alias regressions. Two Windows-only checks were skipped on macOS. Ruff passed.ce5c8865, 240 affected tests passed locally. The Python 3.10 and 3.11 Ready full suites each reported one failure in the Knowledge migration compatibility test for_indexed_directories. Ready CI once5c886551aad393, the affected suites passed 151 tests. A clean Python 3.11 wheel returned the Qwen Case and the packaged fallback body with PyYAML absent. With PyYAML installed, it parsed the displayed description and opened the installed Skill's full body byte for byte. Ruff and the pinned index check passed.dd28a9df, the exact quickstart command ranked the Qwen case first and put the out-of-memory finding at the top of its preview. The returned read command opened the source text. The ranking and no-PyYAML fallback tests passed. All 80 generated index records passed the check against the pinned source revision.findreturnedkernel-writing-guideforcuda graph warmup.collect --relevancestarted a fresh search of packaged material and returnedinferrsi.ascend.skills.ascend_aclgraph_capture_replay. The guide tells callers to assess the new hit before reading its body viaresourcewith itsref,skill_documents.original, andrevision. The read-back for both review queries matched the wheel source byte for byte. The casefind → readflow also worked in the base install. David's review8c0a7bfc, the relevant knowledge search, read, index, installation, module boundary, and plugin suites had 286 passes and 4 skips. Two DSH checks failed with a Node SIGABRT. The same two checks failed in the same way onmainat891aadda. Ruff, diff, and generated index checks passed.find → readas the default discovery workflow for feat(knowledge): unify search across cases, skills, and records #466 in the README, quickstart, and plugins, with the existing commands available for compatibility and fallback. The task: 改进真实优化任务中的材料发现与使用 #380 decision held back the short guidance tested at the time. The Qwen comparison recorded use of retrieved material. Consistent improvements in decision quality and optimization gains remain to be established. Requester decision · Maintainer decisionKnowledge authorship declaration
The PR author confirmed that the earlier search implementation used Claude Opus 5.5 at max effort. A configuration mapping error caused the CLI to label it Opus 5.