Make the agent integrations actually work, and correct the OpenClaw claim - #372
Merged
Conversation
Testing the OpenClaw skill pack against the published v0.15.4 artifact over stdio found that its own example payloads do not work. Nine payloads across four files were rejected outright. Two more files were found by the guard written to stop it recurring. The root cause is one thing, not nine. The skill packs and their recipe docs name the CORE eleven-tool surface but still carry payloads shaped for the legacy alice_vnext_* tools. When the core surface was introduced the tool NAMES were updated and the PAYLOADS were not. The clearest case: every documented explicit commit carried intent: "explicit_remember", which exists only on alice_vnext_commit_memory. alice_memory_commit rejects unknown properties rather than ignoring them, so the flagship example in both packs failed with the opaque tool_request_failed and no reason. Deleting that one key was the difference between a hard failure and status=committed. What was wrong, all reproduced by execution: - intent on alice_memory_commit, in all four skill files. Legacy-only property. - the OpenClaw sprint-output payload sent content/output_type/propose_memory to a core tool. Those belong to alice_vnext_ingest_agent_output, and the payload also omitted the required raw_text. Retargeted at alice_capture, which is review-gated, which is what a generated summary wants anyway. - six proposal examples omitted the required title. - proposal_type is not a shipped field. The four values used map onto real memory_type enum members: project_state, belief, open_loop, contradiction. - all eleven context-pack recipes nested scope and options. The core alice_context_pack takes those fields flat. - three docs sent agents to alice_vnext_confirm_memory and /vnext review queues for lifecycle work. Neither exists on a default server; alice_memory_manage and alice_memory_review do. - two /ABSOLUTE/PATH/TO/AliceBot paths the rename sweep missed. This matters more for Hermes than for OpenClaw. Hermes speaks MCP natively and is the platform this is being promoted on, and its pack was broken the same way. tests/unit/test_agent_facing_payloads_are_callable.py runs every documented payload through _validate_mcp_arguments_against_advertised_schema, the same function the server calls before any handler runs, against tools on the DEFAULT surface. A payload that only works once an operator sets ALICE_MCP_LEGACY_TOOLS=1 is not something to hand an agent unqualified. Adding a JSON block without classifying it fails the suite, so a new example has to be named before it can be checked, and a deleted block cannot leave a phantom entry checking nothing. Verified non-vacuously: reverting the doc edits fails 33 of 106 cases. The guard found two defects I had missed while writing it, an unclassified block and a proposal still missing its title. End to end on the published artifact, with payloads parsed live out of the shipped markdown rather than retyped: ambient commit confirmation_required, explicit commit committed, low-confidence commit review_required, sprint capture imported, project commit committed, and both memories recalled from a NEW session with context_pack memories=1. Every outcome the docs promise now actually occurs. Full unit suite: 4667 passed, 2 skipped.
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
added 2 commits
August 15, 2026 20:36
…the OpenClaw claim
Two corrections, both established by running the real thing rather than reading its docs.
OpenClaw supports MCP. I previously concluded it did not, from a docs read, and that claim
reached the README-adjacent docs and the launch copy. It is wrong. OpenClaw 2026.8.1 carries
@modelcontextprotocol/sdk 1.30.0 as a production dependency with a stdio transport that spawns
child processes. Pointing it at the published artifact:
$ openclaw mcp probe
- alice: 11 tools
$ openclaw mcp doctor
- alice: ok
All eleven core tools, zero diagnostics, no plugin, one config entry under mcp.servers. Tools
arrive prefixed with the server name, so alice_recall is exposed as alice__alice_recall.
Neither skill pack loaded. This is the real defect and it hit Hermes too. Three variants of the
same pack, side by side in OpenClaw's managed skills directory:
DROPPED alice-project-memory-skill.md (what we shipped)
DROPPED SKILL.md, no frontmatter
LOADED SKILL.md + frontmatter modelVisible=true
The drop is silent at every log level: a file not named SKILL.md is never a candidate, and a
file without frontmatter parses to an empty record with zero issues. Hermes uses the same
convention; its own bundled skills are <skill-name>/SKILL.md with name and description in
frontmatter. We had no way to notice either pack was inert.
Both packs are now <skill-name>/SKILL.md with frontmatter, moved with git mv so history
survives. Copied straight out of the repo into OpenClaw with no edits, both now load with
eligible=true and modelVisible=true.
Also fixed, found while verifying: the OpenClaw pack told agents to "create open loops with
alice_open_loops". That tool's actions are list/close/edit/reopen/snooze, so the instruction
could only ever error. Unresolved work is now committed with memory_type "open_loop". Both
packs also tell the agent to read tool names from the host rather than assume the bare form.
README gains the OpenClaw config block, the CLI equivalent, the tool-prefix note, and a skill
pack section that says to copy the directory rather than the file.
tests/unit/test_skill_packs_are_loadable.py enforces the filename, the frontmatter, a
description long enough to be chosen from a system prompt, and directory/name agreement. It
also asserts against the open-loop create instruction, deriving the action enum from the
shipped schema so it revisits itself if create is ever added.
Verified non-vacuously: restoring the old pack shape fails 4 of 11 cases.
Full unit suite: 4678 passed, 2 skipped.
The deployment gate proved "some block in this file demands a client certificate", not "the block serving Alice demands one". Every check in validate_caddyfile is a file-wide substring or directive-set test, which was sound only while the example had exactly one site block. Reproduced against the shipped validator: remove the entire tls/client_auth stanza from the Alice block, leave those same strings in any other block, and all four mTLS checks pass. CI goes green on a configuration that serves the memory API to anyone who can route to it. A second block that proxies straight to 127.0.0.1:8000 with no tls directive also passed. This was harmless while the packaged example had one block and opens the moment a second one exists. That is not hypothetical: the owner is adding a public landing page on the apex while Alice stays behind mutual TLS, which is a hostname split, which is a second block. The gate would have stopped protecting the property it names at exactly the moment the property became non-trivial. validate_caddyfile now splits the file into site blocks by brace depth, finds every block that can reach an Alice upstream, and requires client_auth, mode require_and_verify, and the trust_pool path inside each of them. The file-wide checks are unchanged and still run; this is an additional constraint, not a replacement. A public block is still allowed, and there is a test for it, as long as it cannot reach Alice. That is the supported shape for a landing page on the same host. Verified non-vacuously: the two new mutation tests fail against the previous validator and pass against this one. A third test asserts the stanza constant still appears verbatim in the shipped example, so a reformat cannot turn the mutations into no-ops that pass while removing nothing. Full unit suite: 4682 passed, 2 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You asked me to test the OpenClaw skill. Three things came out of it, each verified by running software rather than reading about it.
1. I was wrong about OpenClaw, and the claim was already public
I previously concluded OpenClaw does not support MCP and that a plugin would be required. That came from reading
docs.openclaw.aiand never opening their source, which was on the machine the whole time.OpenClaw 2026.8.1 ships
@modelcontextprotocol/sdk1.30.0 as a production dependency with a stdio transport. One config entry undermcp.servers, no plugin, zero diagnostics. Tools arrive prefixed, soalice_recallis exposed asalice__alice_recall.The false claim had reached the wiki, the landing page copy, and the announcement drafts, where it sat inside the section arguing for our own candour. All corrected.
2. Neither skill pack was loading, on either host
This is the real defect and it hit Hermes too, which is the platform being promoted on. Three variants of the same pack, side by side in OpenClaw's managed skills directory:
Silent at every log level: a file not named
SKILL.mdis never a candidate, and a file without frontmatter parses to an empty record with zero issues. Hermes uses the same convention. Both packs were inert from the day they were written.Both are now
<skill-name>/SKILL.mdwith frontmatter, moved withgit mv. Copied straight out of the repo into OpenClaw with no edits, both load witheligible=true.3. Every documented payload was rejected by the server
Running the skill's own loop against the published v0.15.4 artifact:
One cause: the packs name the core eleven-tool surface but carry payloads shaped for the legacy
alice_vnext_*tools.intent: "explicit_remember"exists only onalice_vnext_commit_memory, and the core tool rejects unknown properties rather than ignoring them.intenton a core commitcontent/output_type/propose_memoryto a core tool, omits requiredraw_texttitleproposal_typeis not a shipped fieldscope/options; the core tool takes them flatalice_vnext_confirm_memoryand/vnextqueues, neither on a default serveralice_open_loops", whose actions are list/close/edit/reopen/snooze/ABSOLUTE/PATH/TO/AliceBotpaths the rename missed4. Our release gate was blind to a hole the landing page split would open
Found while answering a separate question about putting a public page on the same host. Every check in
validate_caddyfileis a file-wide substring test, so it proved some block demands a client certificate, not that Alice's block does. Reproduced against the shipped validator: strip the wholetls/client_authstanza from the Alice block, leave those strings in any other block, and all four mTLS checks pass. CI goes green on a configuration serving the memory API to anyone.Harmless while the example had one site block. The moment a second one exists it is live, and a hostname split for a landing page is a second block.
Now scoped per block. A public block is still allowed, and there is a test for it, as long as it cannot reach Alice.
Guards
Three new files, each proven non-vacuous by reverting the fix:
test_agent_facing_payloads_are_callable.pytest_skill_packs_are_loadable.pyThe payload guard runs each documented payload through
_validate_mcp_arguments_against_advertised_schema, the same function the server calls before any handler runs, against default-surface tools only. It caught two defects I had missed while writing it.End to end
Payloads parsed live out of the shipped markdown rather than retyped, against the published artifact:
Every outcome the docs promise now occurs, including the graded ones.
Full unit suite: 4682 passed, 2 skipped.