feat(plugin): Stage C — native loading, more manifests, static risk, runtime evidence

[rng1995]

Summary

Stage C and open-gap work for plugin evaluation, stacked on #28. The base is naren/plugin-evaluation-all-tiers; it retargets to main once #28 merges. It closes the remaining gaps from the plugin-evaluation review: more manifest formats, deeper static risk checks, runtime evidence, and native plugin loading for each harness.

Draft for review. Do not merge. Nothing here has been run against a live Harbor run yet; live validation follows review.

What's included

Area	Capability	Gating
Manifests	Codex .codex-plugin/plugin.json, Cursor .cursor-plugin/plugin.json, and Agent Plugins v1 (plugin.json with an agent-plugins.org $schema), in every tier. Precedence when several are present: agent_plugin.yaml > .claude-plugin > Agent Plugins > .codex-plugin > .cursor-plugin. The other manifests are recorded under manifest_declarations, and name/version conflicts are flagged. Components flow into the existing inventory, MCP policy and Tier 3 staging. Codex http_headers are checked together with headers. A root plugin.json that is not UTF-8 or exceeds 1 MiB no longer fails discovery; an additional manifest like that is a MEDIUM plugin_manifest_additional_invalid. An Agent Plugins plugin.json with a JSON syntax error is still auto-detected as a plugin, so the error is reported.	Per-format required-field checks (HIGH/MEDIUM)
Subagent and command privileges	Frontmatter tools / allowed-tools / permissionMode. Flags unrestricted Bash, wildcard tools, bypassPermissions / acceptEdits, and subagents that inherit all tools while write-capable MCP servers are present. MCP denials are matched against Claude Code's mcp__plugin_<plugin>_<server>__* tool names, --read-only=false is not treated as read-only, and tool lists are checked in full, so padding cannot hide unrestricted Bash.	HIGH / MEDIUM, overridable by policy
Hook risk model	Parses events, matchers and handler types (command/http/prompt/agent). Flags broad-matcher auto-approve, context injection, remote code (curl … | sh, including path-prefixed shells, env/sudo/xargs wrappers, intermediate pipe stages, and download-then-run), http hook endpoint policy with a hooks.allowed_urls allowlist compared as parsed URLs (scheme, host, port, and path on a segment boundary), and inline secrets, including credentials embedded in URLs. Matchers are classified without a backtracking regex engine and fail closed when too long or complex. Flat hooks in Cursor and Agent Plugins extensions get the same checks. A hook script that can't be analyzed is a HIGH plugin_hook_script_unanalyzed, and hitting a scan limit is a HIGH plugin_hook_scan_truncated. URL credentials are stripped from finding messages.	CRITICAL / HIGH / MEDIUM / LOW
npm and container CVEs	The opt-in dependency check audits npm exact pins and container image refs with OSV-Scanner, falling back to npm audit --package-lock-only --ignore-scripts, then Grype or Trivy for images. Images from every manifest the plugin ships are covered. Plugin files never reach a scanner directly (synthesized lockfile). The plugin audit never passes silently: no scanner, a failed scanner run (pip-audit included), or a manifest, lockfile or Dockerfile that can't be read within bounds makes the check INCOMPLETE.	Scanner severity
Endpoint DNS/redirect policy	Opt-in validate --resolve-endpoints: resolves MCP (including servers that only an additional manifest declares) and http-hook hosts, flags private or metadata answers, and sends one HEAD request (TLS 1.2 minimum, no redirect following, no credentials). Non-public hosts are never contacted, and hosts with an unexpanded ${VAR} are skipped.	HIGH / MEDIUM
Validator parity	Opt-in --checks claude-validate runs claude plugin validate --strict --json when the CLI is present and reports disagreements.	MEDIUM / LOW / INFO
Hook execution census	hook_census.sh wraps natively loaded hooks and records runs, failures and duration without changing stdin, stdout or the exit code. Natively staged hooks that ran become exercised in coverage. Hooks that weren't staged or loaded are never promoted from census lines.	Advisory
Canary exfiltration	Each task plants a random canary (file plus env var) in both arms. The finding fires when the canary reaches network commands or tools, MCP args, URLs, git, or files outside the workspace, and reports whether the difference is attributable to the plugin. Shell commands are checked per statement, including sh -c/bash -lc/eval payloads, heredocs and argv-list commands, with taint tracked across statements. Whole-environment dumps count; reading one variable by name does not. URLs count only for network, MCP and command tools.	Critical canary_exfiltration
Public MCP proof	Opt-in --probe-mcp: probes the author's URL MCP servers from the host (initialize + tools/list, endpoint policy first), then upgrades status from in-agent evidence: declared → reachable-host → reachable-in-agent → used-successfully. The probe sends only literal declared headers; host environment variables are never expanded unless named with --probe-mcp-env NAME, which can be repeated.	Advisory
Coverage	New states loaded and exercised (precedence exercised > loaded > staged). Subagents and commands observed at runtime count as exercised. The headline counts every evaluated state (staged, loaded, exercised).	Report-only
Native loading + Hermes	--plugin-load {wrapper,native,auto} (default wrapper). The with-plugin arm loads the plugin the way each harness does: claude-code via --plugin-dir (skills, commands, agents, hooks wrapped by the census, .mcp.json, output styles; rules in $CLAUDE_CONFIG_DIR/rules); codex (skills, $CODEX_HOME/AGENTS.md, config.toml mcp_servers); opencode (OPENCODE_CONFIG mcp and instructions, agents, commands); hermes (skills and MCP; rules stay in the wrapper). Codex, Cursor and Agent Plugins manifests are staged through their normalized Claude view. A load census inside the container marks components loaded. Member-skills and no-plugin arms are unchanged, so lift stays comparable. Hermes is added as a Tier 3 agent (containers only). No bypass flags are ever added, and configs carrying them fail closed.	Report-only

All new data is rendered in the JSON, Markdown, HTML and CLI plugin sections (plugin-controlled text is escaped in the CLI report), and documented in docs/plugin-evaluation.mdx, including updated limitations.

Review follow-ups

All 30 comments from the review and the 3 CodeQL alerts are addressed and resolved. The fixes are folded into the table above. The most important:

• --probe-mcp credential exfiltration (High): headers declared as ${VAR} were filled from the host environment and sent to the plugin's own URL. Now only variables named with --probe-mcp-env are expanded.
• hooks.allowed_urls bypass (High): matching was a string prefix, so https://hooks.example.com.evil.net passed. Now parsed URLs are compared.
• Fail-open limits and ReDoS: scan and size limits now produce HIGH or INCOMPLETE results instead of silently passing. Plugin-supplied regexes and the remote-code patterns are bounded; one reviewer case dropped from 18 s to 0.01 s.
• CodeQL: TLS 1.2 minimum for the endpoint HEAD check, and exact-hostname assertions in tests. No open alerts.

Needs live validation (after review)

• The hook census inside real harness hooks: a writable /logs/agent and passthrough of hook decisions.
• Canary placement on every Dockerfile path and in local mode, plus the false-positive rate on real trajectories.
• --probe-mcp against real streamable-HTTP/SSE servers and behind proxies.
• claude plugin validate --json output on current Claude Code builds.
• Native loading on each harness: whether each harness parses the staged config (--plugin-dir in -p mode, $CLAUDE_CONFIG_DIR/rules, the Codex config.toml/AGENTS.md appends, the OpenCode config merge), hooks firing under Harbor's --permission-mode=bypassPermissions, the Hermes install and provider routing, and /skilleval and /logs/agent permissions for non-root agent users. The census proves files are present, not that they were loaded.

Verification

• After the review fixes (63f9f93):
  • ruff check .: passes.
  • Full suite (pytest -n auto): 10,729 passed, 26 skipped, 0 failed.
  • Golden CLI surface: regenerated for --probe-mcp-env.
  • CodeQL: 0 open alerts.
• At submission: the OSS boundary scan and the public benchmark scan passed.

Review fixes

A review of the Stage C code (same design as this PR) turned up correctness and security defects. This push ports the 14 fixes that apply here, and merges the updated naren/plugin-evaluation-all-tiers (#28 with its own ported review fixes) to clear this PR's conflict with its base. Each fix was confirmed against this branch before porting, and each ported regression test fails on the previous commit and passes after.

Area	Ported	Already fixed or not applicable here
Static risk	5: npm audit reports its exit code and stderr when it prints no JSON; npm/image audits are INCOMPLETE past their discovery caps; hooks.allowed_urls entries with userinfo, a query or a fragment match nothing; hook scripts are read under each format's root placeholder (${PLUGIN_ROOT}, ${CURSOR_PLUGIN_ROOT}, relative Cursor scripts, Agent Plugins namespaces); every resolved address must be allowlisted, whatever the DNS answer order	ExternalTool.run env layering, MCP from additional manifests, the additional agent_plugin.yml, and the unreadable root plugin.json were already handled
Runtime	5: --probe-mcp connections are pinned to the policy-checked addresses (answers the DNS-rebinding review comment); grading data (results dirs, generated output, evals source) is kept out of the native Claude Code plugin copy; only the selected manifest's components are staged natively, and hook sources with no staged handlers are reported not_loaded; only the Hermes chat launch gets the native setup prefix; docs	canary sink scoping and the header-variable allowlist were already handled (--probe-mcp-env)
Reports	4: the canary verdict comes from the per-arm rows (no green verdict when the plugin arm leaked); hook-census qualifiers are kept in Markdown and CLI; endpoint rows take their redirect target's classification; docs say where the --probe-mcp private-host allowlist applies	the hook-matcher markup escape and the coverage count were already fixed

🤖 Generated with Claude Code

[rng1995]

Review of Stage C at 16ea9ba: correctness and security findings. Each inline comment was reproduced against a snapshot of this SHA or confirmed by reading the code, and every one has a severity, a failure scenario and a suggested fix.

Highest priority:

• --probe-mcp expands host environment variables into plugin-declared headers and sends them to the plugin's own URL.
• The hooks.allowed_urls prefix match can be bypassed with a look-alike host.

Beyond those, most findings are detection bypasses in plugin_component_risk.py (caps and truncation that fail open, ReDoS), CVE-audit coverage gaps, and canary false negatives and false positives.

e1a0006..28e804a (native loading) was pushed after this SHA. I re-checked every finding against 28e804a, and all still apply. I dropped one docs finding (the hook census never being deployed) because native staging now wires hook_census.sh. The existing CodeQL alerts are not repeated here.

[rng1995]

Live Harbor sample (non-blocking)

A small Harbor sample ran on the equivalent internal code with Claude Code (native and wrapper loading) and Codex (native loading). It confirmed that native loading works (Claude Code reports the plugin, the skill and the MCP server as connected), that hooks run under bypassPermissions, that the canary raised no false positives, and that --probe-mcp reached a public server. It also found follow-ups that apply to this design:

• Markdown reporting crashes when a baseline evaluator score is None.
• The claude plugin validate child process still receives NVIDIA_INFERENCE_KEY, AWS_ACCESS_KEY_ID and similar variables.
• Cursor hook events are staged into Claude's hooks.json without translation.
• Natively loaded skills (<plugin>:<skill>) aren't matched by the component classifier.

🤖 Generated with Claude Code