feat: native OpenClaw gateway plugin (in-process analysis pipeline + dashboard route)#575
Open
zeroaltitude wants to merge 10 commits into
Open
feat: native OpenClaw gateway plugin (in-process analysis pipeline + dashboard route)#575zeroaltitude wants to merge 10 commits into
zeroaltitude wants to merge 10 commits into
Conversation
…pipeline, agent tools, dashboard route Adds .openclaw-plugin/, a real OpenClaw gateway plugin (beyond the existing install.sh skill-symlink integration). The gateway process itself gains: - Agent tools any connected agent can call mid-conversation: understand_list_projects / understand_analyze_project (background job) / understand_status / understand_search / understand_get_node - HTTP routes: /understand-anything project picker that spawns and redirects to the token-protected understand-anything-viewer per project - A self-contained analysis pipeline built entirely on @understand-anything/core's public API (createIgnoreFilter, LanguageRegistry, TreeSitterPlugin, buildFileAnalysisPrompt / parseFileAnalysisResponse, GraphBuilder, validateGraph, saveGraph) with single-turn Anthropic Messages API calls for the LLM enrichment phase — no Claude Code or Task-tool dispatch involved. Output is byte-compatible .ua/knowledge-graph.json. The import-edge resolver handles the TS-ESM convention where "./util.js" refers to util.ts on disk (covered by a regression test; without this every intra-project import edge in a typical TS codebase is silently dropped). Analysis is restricted to a configured project allowlist; concurrency and file-count caps bound LLM cost; the API key comes from plugin config or the gateway process env only. .openclaw-plugin joins the pnpm workspace so it can depend on @understand-anything/core via workspace:*.
…agent tools Current OpenClaw refuses to register plugin agent tools unless the manifest declares them up front in contracts.tools (registry.ts:617 "plugin must declare contracts.tools before registering agent tools"). Verified via `openclaw plugins inspect understand-anything --runtime` against an isolated state dir: before this, activate() hit 5 error diagnostics and registered 0 tools; after, all 5 tools + both dashboard routes register with 0 errors.
An independent GPT-5.6-Sol review of the plugin (report in ~/reports/understand-anything-plugin-review-2026-07-12.md, not part of this repo) found 4 must-fix and 4 should-fix issues, spot-checked and confirmed against the source before addressing: Must-fix: - walk.ts: walkProject followed symlinks with no containment check or cycle detection. A symlink under an allowed project root (not covered by the default ignore patterns) could read files outside the projects allowlist — the plugin's entire security boundary — and a symlink cycle could recurse forever and crash the whole gateway process (this runs in-process, not in a subprocess). Fixed via realpathSync containment checks plus a visited-dirs set seeded with the root itself (a naive per-child-only visited set still let a *self*-referential top-level symlink walk root's contents one extra time before the guard caught it). Added src/__tests__/walk.test.ts covering escape, deep cycle, and self-cycle cases with real filesystem symlinks. - dashboard-route.ts: getOrStartViewer only wrote to the `viewers` map once spawn() produced its dashboard URL (up to 15s later), so two concurrent requests for the same project's dashboard both saw a cache miss and both spawned a viewer — the loser leaked as an unreferenced, never-killed process. Fixed by storing the in-flight *promise* in the map synchronously before spawn() returns. Verified live (not just reasoned about): fired two concurrent requests against the built module and confirmed both resolved to the identical viewer instance (same port, same token). - dashboard-route.ts / index.ts: no lifecycle management at all for spawned understand-anything-viewer processes — no shutdown hook, no idle eviction, no cap. Every project dashboard opened once left a Node child process bound to a port for the life of the gateway, and a plugin/gateway reload orphaned all of them (module state resets, PIDs become unreachable). Added shutdownAllViewers() wired to the gateway_stop hook, plus a 30-minute idle-eviction sweep (unref'd so it can't itself keep the process alive). - llm.ts: no timeout on the Anthropic HTTPS call. A single stalled TCP connection never settled, permanently consuming a mapWithConcurrency worker slot and wedging that analysis job in "running" state forever — the only recovery was a full gateway restart, since understand_analyze_project refuses to start a new job while one is "running". Added a 60s request timeout via the `timeout` socket option + a `timeout` event handler that destroys the request and rejects cleanly. Should-fix: - index.ts: getGraph's loadGraph/JSON.parse path had no try/catch. saveGraph (in @understand-anything/core) does a plain writeFileSync with no temp-file+rename, and the gateway now serves reads continuously while a background analyzeProject() job can write at any moment — a read landing mid-write threw an uncaught exception. Now falls back to the previously cached graph on parse failure instead of propagating. - llm.ts: resolveAnthropicApiKey defaulted to the "main" agent's auth-profile path when OPENCLAW_AGENT_DIR wasn't set. A gateway hosting multiple agents with separate homes could silently read the wrong agent's key/account. The auth-profile fallback now only applies when the host has actually told us which agent this is. - index.ts: nodeBrief did `n.tags.length` unconditionally despite graphs being loaded with `validate: false` specifically to tolerate off-schema data. Guarded with an Array.isArray check. - dashboard-route.ts: /understand-anything/open with no `project` query param at all passed validation (Number(null) === 0) and silently opened project index 0. Now requires the param to be present. Nice-to-have also applied: pipeline.ts called builder.build() twice (once to preview layer node IDs, once for the persisted graph) — build() is a pure, deterministic snapshot, so this was a needless double array-copy on potentially thousands of nodes/edges. Now built once and mutated in place. Full pipeline re-verified end-to-end after these changes (real Anthropic key, real project, matching node/edge counts, 0 warnings, no regression). 17/17 unit tests pass (13 existing + 4 new).
Adds a floating "Ask" chat widget to the dashboard, backed by a new POST /ask.json endpoint — the missing piece between "generate a graph" and "actually talk to it," without leaving the browser for a CLI/chat session. Architecture, deliberately not a fork of understand-anything-viewer: that package's whole reason to exist is staying LLM-free for team-sharing without an API key, so it's never modified. Instead: - src/interactive-server.ts is a new, separate server (spawned as its own subprocess, same pattern as the existing viewer proxy) that reuses the viewer's static dashboard build and JSON API routes read-only, injects a small vanilla-JS widget script into index.html, and adds the /ask.json endpoint. Same token/security model as the existing endpoints throughout. - src/ask.ts is the actual Q&A logic: SearchEngine finds graph nodes relevant to the question, a bounded number of their source files are read for grounding, and the same llm.ts caller used for analysis answers using only that context. Mirrors upstream's /understand-chat skill, just reachable from the browser. - src/ask-widget.js is the floating chat UI: vanilla JS/CSS, no build step, so it's independent of the dashboard's React app and build pipeline. - dashboard-route.ts now branches on whether an API key is resolvable (same resolution order the analysis pipeline already uses): with a key, serves the interactive server; without one, falls back to the plain viewer exactly as before — verified both paths live, including that the plain-viewer fallback still returns the correct redirect with no widget injected. Verified end-to-end with a real Anthropic key against a real analyzed project: the widget serves, /ask.json answers a real question accurately (correctly identified and cited the actual ingestion code paths across multiple files), and the existing graph/file-content API is unaffected. 3 new unit tests for ask.ts's grounding logic (no-graph case, node-matching, snippet inclusion) — 20/20 total tests pass.
…board Two picker-page additions: 1. Unanalyzed projects get an "Understand this project" button (POST /understand-anything/analyze?project=<idx>) instead of a plain "call the tool" note. Shares one job-tracking path with the understand_analyze_project tool via a new startAnalysis() function extracted from the tool body — not a second, divergent implementation. The picker auto-refreshes (plain <meta refresh>, no JS) while any job is running, and shows a "Retry analysis" button if one failed. 2. An "Add a project" form (gated behind a new allowAddProject config flag, default false) lets the dashboard register a new project at runtime by GitHub URL or local path, then immediately starts analyzing it. src/project-store.ts is the new ProjectStore: combines config-declared projects (fixed) with dynamically-added ones (persisted to dynamic-projects.json under ~/.local/share/understand-anything-plugin, so they survive a restart) into one index-stable list — new entries only ever append, so existing tool/dashboard references by index never break. GitHub URL handling only recognizes an actual https://github.com/... or git@github.com:... URL — deliberately NOT a bare "owner/repo" shorthand (ambiguous with a genuine relative local path) or any non-GitHub git/ file:// URL (which could otherwise smuggle an arbitrary local path or host into a "clone", defeating the projects allowlist this feature sits inside). Clones shallow (--depth 1) with a 3-minute timeout via a real array-args spawn() (no shell, no injection risk). allowAddProject defaults to false and is documented as a real security tradeoff in the README: the projects allowlist is otherwise the only way to expand what this plugin can read/analyze/serve, and this setting relaxes that for anyone who can reach the dashboard route. Verified end-to-end with a real live GitHub clone (not just unit tests): activated the full plugin, clicked "Understand this project" on a fresh unanalyzed project (job started, status tool confirmed "running"), then added octocat/Hello-World by its real GitHub URL — genuinely cloned to ~/.local/share/understand-anything-plugin/clones/octocat-Hello-World, appeared in both the picker and understand_list_projects at the correct index, and auto-started analysis. Also verified the allowAddProject: false default correctly hides the form and rejects the POST endpoint. 6 new unit tests for ProjectStore (persistence round-trip, duplicate handling, non-existent-path rejection, GitHub shorthand/non-GitHub-URL rejection) — 26/26 total tests pass.
Three additions, plus a real bug fix uncovered along the way: 1. Re-analyze button: already-analyzed projects on the picker page now get a "Re-analyze" button (analysis was always a full fresh re-run, just no UI trigger existed for the already-analyzed case) — this is how a project's tours/summaries reset to whatever the pipeline currently produces as it improves. 2. Auto-generated tours at analysis time (src/tour-generation.ts, src/tour-store.ts): a free, deterministic module-walkthrough tour (wraps core's generateHeuristicTour, zero LLM cost) is synced into the standard graph.tour field — upstream's schema only supports one tour, so this keeps the existing Learn persona/LearnPanel working with zero changes. A second, LLM-narrated code-review tour ranks nodes by complexity + how central they are in the dependency graph (fan-in/fan-out), then asks the model to explain *why* a reviewer should care about each, not just what the code does. Both persist to a plugin-owned .ua/tours.json sidecar (upstream's schema has no room for multiple named tours). Lifecycle/request-flow tours are explicitly deferred — that needs call-graph/endpoint traversal work, a separate research task. 3. Custom tour generation (src/custom-tour.ts) + a new Tours widget (src/tours-widget.js, same vanilla-JS/no-build-step pattern as the Ask widget): select node(s) in the live React Flow canvas — read via a MutationObserver on .react-flow__node.selected, zero patches to the dashboard's React source — type a prompt, and get back a tour scoped to just those nodes and that request. New endpoints on interactive-server.ts: GET /tours.json (list) and POST /generate-tour.json (generate + persist), same per-instance token model as the existing endpoints. Bug fix found while testing tour generation live: llm.ts's createLlmCaller assumed content[0] was always the text block, but extended thinking (on by default for some models/accounts) puts a "thinking" block first, with the actual response in a later block — every LLM call in the plugin (analysis, Ask, tours) shared this latent bug, it just hadn't been triggered before now. Fixed to find the first block with type "text" instead of blindly indexing [0]. Added 4 regression tests mocking node:https (common case, thinking-first case, no-text-block case, API-error case). Verified end-to-end with real live Anthropic calls, not just unit tests: analyzed a real project (module + code-review tours both generated correctly, zero warnings after the llm.ts fix), started the interactive server, confirmed both widget scripts inject, hit GET /tours.json and saw both auto-tours, then POST /generate-tour.json with real node ids and a real prompt — got back an accurate 5-step tour correctly tracing the actual CLI entry point through to post-ingestion cleanup, persisted alongside the other two tours. Confirmed all validation/security paths (empty nodeIds, empty prompt, missing token → 403). 14 new tests (custom-tour, tour-generation, tour-store, llm regression) — 44/44 total tests pass.
Formalizes an on-disk contract that already existed informally: the understand-diff skill has always read/written <ua-dir>/diff-overlay.json in this exact shape (SKILL.md step 8), and the dashboard's store/GraphView already consume changedNodeIds/affectedNodeIds for the diff-mode overlay UI — but core had no first-class type or persistence pair for it, unlike the sibling domain-graph.json (saveDomainGraph/loadDomainGraph). - DiffOverlay type in types.ts. - saveDiffOverlay/loadDiffOverlay in persistence/index.ts, mirroring the domain-graph save/load pattern exactly (same directory resolution, same file-path sanitization so absolute paths never leak to disk — extracted the per-path sanitization logic out of sanitiseFilePaths into a reusable sanitiseFilePath so both the node-array and plain-string-array shapes can share it, no behavior change to the existing function). - computeDiffOverlay(graph, changedFiles, baseBranch?) in the new diff-overlay.ts: pure, deterministic graph traversal — maps changed file paths to their file/function/class nodes (graph-builder stamps the same filePath on all three), then walks one hop of edges in both directions to find the blast radius, excluding the changed nodes themselves. No LLM involved, mirrors the skill's own documented algorithm (grep node IDs in edges, 1-hop) exactly. 11 new tests (5 persistence round-trip/sanitization, 6 traversal logic). Full suite: 942/942 tests pass (931 existing + 11 new).
…ntrol) Resolves changed files via `gh pr diff` or `git diff <base>...HEAD` (falling back to uncommitted working-tree changes), computes the blast radius against the persisted graph with core's computeDiffOverlay (pure, no LLM), then asks the model to narrate a tour grounded only in the changed + affected nodes. Exposed as both the understand_analyze_pr agent tool and a POST /generate-pr-tour.json endpoint, with a matching input + button in the Tours widget. Node lists sent to the model are capped (40 changed / 20 affected) so the required output stays bounded on large diffs — confirmed via live E2E testing against a real branch diff that an uncapped prompt plus the previous 2000 max_tokens ceiling produced a mid-JSON truncation (stop_reason: max_tokens). Raised the budget to 8000 and added the caps together, then re-verified a 7-step walkthrough generates cleanly against the same real diff. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ry script
Ask (src/ask.ts) previously had zero notion of what the user is currently
looking at — pure free-text SearchEngine relevance. Custom-tour generation
already scoped to an explicit node selection (read via a private
MutationObserver on .react-flow__node.selected in tours-widget.js), so a
question asked while a specific file was selected in the canvas could ignore
that file entirely if the question's wording didn't happen to match it.
Extracted the selection-reading logic into src/selection.js, a single shared
discovery script (window.uaSelection.get()/.subscribe()) loaded before both
widgets, so there's exactly one MutationObserver instead of two independently
watching the same DOM. Both ask-widget.js and tours-widget.js now read from
it instead of keeping private copies.
askAboutProject gained an optional selectedNodeIds param: selected nodes are
always included as primary grounding context (deduped against SearchEngine's
matches), ahead of and independent from whatever the question text happens to
match. The Ask panel also shows a "Grounded in: ..." bar so it's visible
whether a selection was picked up.
Verified live against a real analyzed project: the same vague question ("what
does this file do?") answered about whatever file ranked highest generically
without a selection, and correctly answered about the selected file
(src/db.ts) once one was set — despite an identical question in both cases.
3 new tests (selection grounding, selected+matched dedup, unknown node id
ignored) — 52/52 total pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
7 tasks
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e342d61c0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| function loadTours() { | ||
| return authedFetch("/tours.json").then(function (res) { return res.json(); }).then(function (data) { return data.tours || []; }); |
There was a problem hiding this comment.
In the interactive server, /tours.json is protected but is not in TOKEN_VIA_HEADER, so it only accepts ?token= while this call sends the token only as X-Ask-Token. With a valid dashboard session, opening the Tours panel therefore gets a 403 response and silently renders an empty tour list, hiding the module/code-review/custom tours. Add the query token here or allow the header for /tours.json.
Useful? React with 👍 / 👎.
| (function () { | ||
| "use strict"; | ||
|
|
||
| var TOKEN = new URLSearchParams(window.location.search).get("token") || ""; |
There was a problem hiding this comment.
The dashboard app persists the access token in sessionStorage and removes ?token= from the URL, but the injected widgets only read window.location.search. After a normal refresh/bookmark of that cleaned dashboard URL, or after entering the token through the existing TokenGate, the core dashboard still loads while Ask (and the same pattern in tours-widget.js) sees an empty token and returns from init, so the interactive controls disappear. Fall back to the same stored token or inject it separately.
Useful? React with 👍 / 👎.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
.openclaw-plugin/— a native OpenClaw gateway plugin that runs the Understand-Anything knowledge-graph pipeline inside the gateway process itself, as a deeper alternative to the existinginstall.sh openclawskill-symlink integration (which is untouched and keeps working).Why this exists: the skill-symlink integration only runs the analysis pipeline when a Task-tool-capable coding agent drives it step-by-step through the 848-line
/understandskill. This plugin runs the same pipeline as ordinary backend code attached to the gateway's own lifecycle instead — any agent connected to that gateway getsunderstand_*tools mid-conversation, plus a persistent, LLM-backed dashboard, with no CLI-specific dispatch required.Agent tools:
understand_list_projects,understand_analyze_project,understand_status,understand_search,understand_get_node,understand_analyze_pr.Dashboard: an interactive Ask chat panel, auto-generated + custom + PR-walkthrough tours, click-to-analyze, add-project-by-URL — all layered additively on top of the existing zero-LLM
understand-anything-viewer(never modified — its whole reason to exist is staying LLM-free for team-sharing).This PR was built incrementally; below is what each piece does and why, condensed from the original review history (previously on #569, closed as part of a branch-naming cleanup — this PR is that same work, unchanged, just correctly named).
What's in here
1. Core plugin scaffold.
src/pipeline.tsruns analysis using only@understand-anything/core's already-public API (tree-sitter structure, LLM file/project-summary prompts,GraphBuilder→saveGraph) — no new parsing/prompt logic, just new orchestration and a new host. Output is byte-compatible with the rest of the ecosystem (same.ua/knowledge-graph.jsonshape, same dashboard renders it). Analysis/serving are restricted to a configuredprojectsallowlist; the Anthropic API key never touches disk.2. Ask panel (
src/ask.ts,src/ask-widget.js,POST /ask.json). Lets you ask the dashboard a question directly instead of needing a CLI/chat session.SearchEnginefinds the graph nodes most relevant to the question, reads a bounded number of their source files, and a single LLM call answers using only that context — same idea as upstream's/understand-chatskill, reachable from the browser.3. Click-to-analyze + add-project-from-dashboard. Unanalyzed projects get an "Understand this project" button on the picker page. Behind a new
allowAddProjectconfig flag (default off — a deliberate security tradeoff, since theprojectsallowlist is otherwise the only thing bounding what the plugin can read), you can also paste a GitHub URL or local path to register + analyze a new project without editing config.4. Tours: auto-generated, custom, and re-analyze. Every analyzed project gets a deterministic module walkthrough (synced into upstream's own
graph.tourfield, so the Learn persona keeps working unmodified) and an LLM-narrated code-review tour ranking the riskiest/most-central files. You can also select node(s) in the live graph canvas and type a prompt to generate a custom tour scoped to just that selection. A re-analyze button was added for already-analyzed projects (analysis is always a full fresh re-run).5. PR/diff walkthroughs (
understand_analyze_prtool,POST /generate-pr-tour.json). Answers "can this framework explain a change, not just a static codebase?" Resolves a PR's (or branch diff's) changed files viagh pr diff/git diff <base>...HEAD, computes the 1-hop blast radius against the already-analyzed graph using core's purecomputeDiffOverlay(from #574 — no LLM involved in that step), then narrates a walkthrough grounded strictly in the changed + affected nodes. Node lists sent to the model are capped so large diffs stay within budget instead of truncating mid-response.6. Ask + Tours share one live-selection mechanism. This is the piece that prompted the cleanup — worth calling out clearly since it's easy to miss buried in commit history:
MutationObserveron.react-flow__node.selected). The Ask panel had no equivalent — it answered purely from the question's own text viaSearchEngine, with zero awareness of what you were actually looking at. Caught this live: selected a file via the dashboard's Files breadcrumb, asked a question, and the answer never mentioned the selected file at all.tours-widget.jsinto a new sharedsrc/selection.js(window.uaSelection.get()/.subscribe()) — oneMutationObserverinstead of two independently watching the same DOM. Both widgets now read from it.askAboutProjectgained an optionalselectedNodeIdsparam: selected nodes are always included as primary grounding context, ahead of and independent from whatever the question text happens to match. The Ask panel now shows a "Grounded in: ..." bar so it's visible whether a selection was picked up.Test plan
pnpm --filter @understand-anything/openclaw-plugin build(tsc) — cleanpnpm --filter @understand-anything/openclaw-plugin test— 52/52 passpnpm --filter @understand-anything/core test— 942/942 passloaded, all 6 tools + HTTP routes registered,understand_analyze_prconfirmed callable post-restartNotes for reviewers
DiffOverlaypiece) for the PR/diff-walkthrough feature.