feat(plugin): Add Plugin evaluation support across all tiers

[chrisknvidia]

Summary

This PR adds plugin evaluation to every tier of SkillEvaluator. It takes a plugin in any of five manifest formats (agent_plugin.yaml, Claude Code, Codex, Cursor or Agent Plugins v1), checks its structure and security statically (Tier 1), checks it for duplication (Tier 2), and measures its behavior with live agents (Tier 3), loading it through a generated wrapper skill or natively in Claude Code, Codex and OpenCode. The reports say plainly what was and was not demonstrated.

Files staged ≠ components loaded ≠ behavior verified. Every plugin report separates what was inventoried, what was staged into a run, what the harness reported as loaded, what was observed in agent traces, and what was not evaluated at all. Missing evidence narrows the claim; it is never silently dropped.

Full documentation: docs/plugin-evaluation.mdx.

How plugin evaluation works

flowchart LR
    P["Plugin<br/>agent_plugin.yaml · Claude Code · Codex · Cursor · Agent Plugins v1<br/>skills · rules · MCP · hooks · agents · commands ·<br/>LSP · monitors · settings · styles"]

    subgraph T1["Tier 1 · Static & security (gating)"]
        direction TB
        T1a["Manifest checks per format<br/>+ component inventory"]
        T1b["Dependency classification<br/>provided / referenced / missing / external / unresolved"]
        T1c["MCP policy · pinning · endpoints<br/>(opt-in DNS and redirect checks)"]
        T1d["Hook risk · agent, command and skill privileges<br/>LSP, monitor and settings checks"]
        T1e["Whole-tree scans · bundled-skill quality/lint/version<br/>opt-in CVE audit: Python, npm, containers"]
        T1f["Static context-cost estimate"]
    end

    subgraph T2["Tier 2 · Deduplication (advisory)"]
        direction TB
        T2a["A · duplicate refs"]
        T2b["C-intra · bundled-skill overlap"]
        T2c["C-inter · skills vs local catalog"]
        T2d["B · plugin vs other plugins<br/>(optional LLM verdict)"]
    end

    subgraph T3["Tier 3 · Live agent evaluation (Harbor)"]
        direction TB
        L["Plugin loading<br/>wrapper (default): generated skill + members + rules + MCP<br/>native: claude-code · codex · opencode adapters"]
        A1["With plugin<br/>load census · hook census"]
        A2["Without plugin"]
        A3["Member skills only<br/>(sum of parts)"]
        G["Graders<br/>security incl. canary exfiltration · execution ·<br/>judges (N/A without a reference)"]
        S["Plugin signals (advisory)<br/>activations · routing · tool selection · arguments ·<br/>MCP outcomes · order · handoff · conflict"]
        M["MCP proof (opt-in)<br/>host probe + in-agent calls"]
        ST["Statistics<br/>paired bootstrap CI · pass@k / pass^k ·<br/>cost per success · token efficiency · context delta"]
    end

    R["Reports<br/>JSON · Markdown · HTML · SARIF · CLI<br/>coverage: staged / loaded / exercised<br/>INCOMPLETE / INCONCLUSIVE · plugin BENCHMARK.md"]

    P --> T1 --> T2 --> T3 --> R
    L --> A1
    A1 --> G
    A2 --> G
    A3 --> G
    G --> S --> ST
    S --> M

Loading

What each tier supports

Tier	Capability	Gating
1	Manifests: agent_plugin.yaml and the Claude Code, Agent Plugins v1, Codex and Cursor plugin.json formats, selected in that order, with root-bounded, no-follow discovery and each client's field checks. Other manifests found are recorded and their components checked. Opt-in claude-validate compares the verdict with claude plugin validate for Claude Code plugins (advisory)	Blocking on invalid or unsafe input; MEDIUM plugin_manifest_conflict on a name or version mismatch
1	Component inventory: skills, rules, MCP, hooks (frontmatter hooks too), subagents, commands, LSP, output styles, monitors, settings, Codex apps and Agent Plugins extensions, each with a support level and per-component finding counts	Missing, escaping, unsafe or invalid declared paths block (HIGH)
1	Dependency classification: provided / referenced / missing / external / unresolved, with fail-closed repository identity and validate --repo-root. Claude Code dependencies resolve against the plugin's marketplace.json	Only missing blocks (HIGH plugin_dependency_missing); refs it cannot check are MEDIUM
1	MCP static policy for every mcpServers form (inline, path, array, root .mcp.json), read with each client's ${VAR} expansion	CRITICAL and HIGH block; Tier 3 staging re-checks them
1	Pinning of package runners and images (npx, uvx, pnpm dlx, docker run, ...) through launch wrappers, in MCP servers and command hooks, with a pinning ratio	HIGH mcp_command_floating_version for a moving tag; MEDIUM mcp_unpinned_package otherwise
1	Bypass flags and overrides in MCP servers, hooks, LSP servers, monitors and settings (--dangerously-skip-permissions, bypassPermissions, Codex -a never, auto-approve, LD_PRELOAD, proxy and base-URL redirects, shipped .env)	HIGH or MEDIUM, by type
1	Endpoint policy for metadata, private, loopback and link-local hosts (encoded IPs too), with an mcp.allowed_private_hosts allowlist. Opt-in --resolve-endpoints adds DNS and one HEAD check	HIGH (metadata, never allowlisted) or MEDIUM
1	Hook risk per handler, scripts included: remote code (curl … | sh, download-then-run), inline secrets, broad auto-approval, HTTP endpoints (hooks.allowed_urls), context injection	CRITICAL to LOW; an unanalyzable script is HIGH
1	Subagent, command and skill privileges (tools, allowed-tools, permissionMode)	HIGH: bypassPermissions, or allowed-tools that pre-approve any Bash. MEDIUM: wildcards, acceptEdits/auto, subagents that inherit every tool beside a write-capable MCP server. LOW: a subagent's unrestricted Bash
1	Whole-plugin-tree security, PII, code-risk, secrets, license, hygiene and Unicode scans, plus quality/lint/version on every bundled skill	Existing Tier 1 gates
1	Opt-in dependency audit of declared exact pins, installing nothing: Python (pip-audit --no-deps --disable-pip), npm (lockfiles, package.json, MCP runner packages) and container images (MCP and Dockerfiles), with OSV-Scanner, npm audit, Grype or Trivy	Advisory severity. Pins that pip-audit skips or the npm registry lacks are MEDIUM dependency-not-audited; floating versions are INFO; no scanner or an unreadable file is INCOMPLETE
1	Static always-on vs on-demand context-cost estimate, per harness and load mode	Report-only
2	Check A (duplicate refs), C-intra (bundled-skill overlap), and the local-catalog checks C-inter (skills vs catalog) and B (plugin vs other plugins, optional --llm verdict). Catalog v2 stores plugin entries; v1 catalogs still load	Advisory; unsafe input blocks
3	Three arms: with plugin, without plugin, member skills only. --lift-mode effectiveness|integration|both, with the Integration evidence gate (cross_component, expected_skills)	Gates only with --block-on-agent-eval
3	--plugin-load wrapper|native|auto (default wrapper) for the with-plugin arm only: native adapters for claude-code (--plugin-dir), codex and opencode; Hermes (experimental) uses the wrapper. A per-trial load census lists staged components; Claude Code's init event marks them loaded or not_loaded	Report-only. native refuses components with a permission bypass; a native run with no load census, or whose plugin Claude Code never loaded, is INCOMPLETE
3	Hook census: native command hooks run through hook_census.sh, which records runs, denials and start failures and passes input, output and exit code through	Advisory; marks hooks exercised
3	Canary exfiltration: a random decoy credential (file and env var) in every arm, traced through shell statements, network and MCP tools, URLs, git, and files written outside the workspace	Critical canary_exfiltration; plugin-attributable when the plugin arm leaks more often than the baseline
3	Runtime security: credential-store reads and protected-file writes, matched as normalized whole paths, subagent and child-agent calls included	Critical scores 0.0
3	MCP proof (opt-in --probe-mcp): a policy-checked host initialize and tools/list for URL servers, with no host credentials unless a variable is named with --probe-mcp-env, then updated from the agent's own calls	Advisory
3	Paired case-bootstrap 95% CI on Effectiveness and Integration lift, with a precision flag. pass^k, cost per success, token efficiency, measured first-turn context delta	Report-only; Integration becomes INCONCLUSIVE when the CI crosses a ±0.05 band edge, fewer than 5 cases pair, or coverage is incomplete
3	Plugin signals (dataset-driven): activations, routing and tool-selection P/R/F1 with decoys, argument checks, MCP call outcomes by server and tool, order, handoff, conflict probes, activation coverage	Advisory, report-only
3	Judges without a reference (ground_truth / expected_behavior) are N/A, not a fabricated 1.0	See reviewer notes
All	Reports: plugin sections in JSON, Markdown, HTML, SARIF and the CLI, including plugin loading, hook census, canary and MCP proof. Tier 3 coverage per component (staged, loaded, not_loaded, exercised, ...), headlined as "N components not staged"; explicit INCOMPLETE and INCONCLUSIVE; plugin_provenance.json re-read when a report is rebuilt; plugin BENCHMARK.md card	—

Reviewer notes: behavior changes to be aware of

• N/A judges. Accuracy, goal accuracy and behavior check no longer return 1.0 when the case has no reference. They are excluded from the overall score and from lift, so a run where no case has ground_truth now gets a NEUTRAL verdict at best, because its Correctness evidence is missing. reward.json stays numeric-only for Harbor, and the N/A markers travel in the sidecar.
• One lift basis, skills included. The arm without the skill or plugin scores skill_execution and skill_efficiency as N/A, and every lift compares its two arms case by case on the dimensions both scored. Activation alone no longer earns lift, so Skill Lift values change for skill runs too.
• Tier 3 gate. With --block-on-agent-eval, a Tier 3 FAIL verdict now fails validate, and so does a confirmed Skill Lift regression (lift at or below −0.10 with its whole interval below zero). NEUTRAL never does.
• Scores can move for skill runs too. Judges see the full body of each file write, within the evidence budget, instead of its first 200 characters. Runtime security checks more credential stores, protected files and subagent calls, and treats curl … | sh and forced git push as critical.
• New blocking Tier 1 findings for plugins: the CRITICAL and HIGH findings of the manifest, component-path, dependency, MCP, endpoint, bypass-flag, hook-risk and privilege checks above. Tier 3 staging fails closed on the same MCP findings.
• Bundled skills are gated. Quality, lint and version now run on each bundled skill, so a weak bundled skill can fail a plugin that used to pass.
• Integration verdict (advisory). It reads the whole interval against the ±0.05 band and is inconclusive when the interval crosses a band edge, fewer than 5 cases pair, or per-case coverage is incomplete. The point estimate's band is kept as point_verdict.
• The dependency audit no longer installs anything. It audits exact pins with --no-deps --disable-pip, so transitive coverage from requirements files is traded for not executing untrusted build code. Python advisories take their severity from the public OSV record (GitHub severity, then CVSS) rather than defaulting to HIGH.

Public adaptations and exclusions

• Public GitHub/Git references and same-repository offline resolution only. Nothing internal is included: no GitLab, P4, managed execution, private providers or endpoints, credentials, telemetry, provider-registry MCP resolution, or remote vector DB.
• Tier 1 plugin checks run offline unless you opt in: --resolve-endpoints adds DNS and redirect checks, and the dependency audit uses its scanners' advisory and registry lookups. Tier 2 catalogs are local files.
• Not yet supported, and reported as such rather than hidden:
  • runtime tests of LSP servers and monitors: native Claude Code stages LSP servers, but nothing confirms they ran, and monitors are never staged;
  • native-load confirmation outside Claude Code: Codex and OpenCode components stay listed (found, not confirmed by the harness); Hermes has no native plugin path, and no end-to-end Hermes plugin run has been verified;
  • process-level evidence: runtime security and the canary read the agent's tool calls and the files they wrote, so a hook or MCP server process acting on its own is not observed;
  • an equivalent of internal's provider-registry MCP proof: provider-only MCP servers are never resolved or probed (the run is INCOMPLETE), and --probe-mcp covers URL servers only;
  • MCP env, headers and ${user_config.*} values that the with-plugin arm cannot apply (only native Claude Code applies them); such runs are INCOMPLETE;
  • Codex apps and Agent Plugins extensions, which are inventoried only.

Included pull requests

• #171 (Stage C): native plugin loading, the Codex, Cursor and Agent Plugins v1 manifests, static risk checks for hooks, privileges, LSP, monitors, settings and output styles, runtime evidence (load and hook census, canary), npm and container CVE audits, and the MCP proof.
• #180 and #181: fixes from two rounds of code review of this PR.
• The branch tracks main through v0.5.0, including the Harbor 0.24.0 upgrade from #83.

Verification

• uv run pytest -q -n 4 (with main through v0.5.0 merged): 17,480 passed, 29 skipped, 0 failed
• uv run ruff check .: passed
• scripts/check_oss_boundary.py and scripts/ci/check_public_benchmarks.py --require-files tests/golden: passed
• fern check: 0 errors

A sample live Harbor run on the internal counterpart of this work (MR !280) passed for Claude Code and Codex with native loading (results). That run predates the Harbor 0.24.0 upgrade, which passed its own live parity runs in #83; a live plugin run on Harbor 0.24.0 is still to do.

Verification also covers focused regressions per feature (including symlink, escape and oversize cases) and packaging and boundary checks.

Restoration note (from the original description)

This PR replaces #17, which GitHub permanently closed after the default-branch history was consolidated and the original head branch was deleted and recreated. The original discussion and commit history remain available on #17.

🤖 Generated with Claude Code

[rng1995]

The plugin evaluation implementation and its follow-up hardening are well covered, but I found one small diff-hygiene issue to clean up.

[rng1995]

Approved after the EOF diff-hygiene issue was fixed in 3965061, the review thread was resolved, and the complete GitHub check matrix passed.

[rng1995]

Fresh review found two actionable security issues in the plugin evaluation staging path. The previous EOF-hygiene thread is already fixed and resolved. I will address these findings while updating the branch from current main.

[rng1995]

Approved after fresh security review and remediation: all review threads are resolved, the branch is conflict-free against current main, the complete local suite passes (3,999 passed, 21 skipped, 3 deselected), and all 15 GitHub checks pass including Windows, packaging, DCO, and security scans.

[mohgupta-ship-it]

Posted by Codex on behalf of Mohit.

Request changes: plugin provenance is persisted through a symlink-following path after the long-running evaluation. This is a medium output-integrity risk for a same-privilege actor able to modify the selected results location. The inline note describes a no-follow, atomic remediation and the regression coverage needed.

[mohgupta-ship-it]

Codex review on behalf of Mohit — REQUEST CHANGES

Critical sandbox/path-integrity blocker. Two safe reproductions show plugin-controlled Git symlinks can cause host-readable content to be staged before Docker isolation: (1) a member evals/evals.* symlink is accepted through find_eval_file(...).exists(), parsed, and copied into task inputs before any symlink validation; (2) a repo-root skills or rules symlink is resolved before containment, making the external target the trusted root.

This is a pre-sandbox host-file disclosure path, not only a race. Reject links/reparse points and mount crossings before resolution; use descriptor-anchored no-follow reads/copies for member datasets (new and legacy layouts); preserve the lexical clone-root boundary for canonical refs; and add regressions for both attack paths.

Secondary integrity issue: fail closed when the Git-origin slug is unavailable — the current fallback can mark a foreign reference as fully evaluated.

[rng1995]

@chrisknvidia Gentle ping when you have a chance: there are still three unresolved review threads on this PR. Please take a look and update the branch or reply on the threads where you disagree.

[rng1995]

Updated in f3aefb9: merged current main and resolved the conflicts, hardened provenance/canonical-ref/member-dataset staging, and fixed the custom-only sum-of-parts report path uncovered during the merge review. All addressed review threads are resolved. Local verification covered the full suite and focused security/report regressions; Ruff and git diff --check pass. All GitHub checks, including Python 3.12/3.13, DCO, CodeQL, packaging, Windows Tier 2, and macOS Tier 3, are green.