[luv-88] feat: convention-based policy auto-discovery

[NiveditJain]

Summary

• Add git-hooks-like auto-discovery: drop *policies.{js,mjs,ts} files into .failproofai/policies/ at project or user level and they're loaded automatically — no --custom flag or config changes needed
• Convention hooks registered with convention/ prefix for PostHog telemetry (convention_policies_loaded event, is_convention_policy flag)
• listHooks() displays discovered convention policies
• CLI help updated with CONVENTION POLICIES section
• 6 new unit tests, 16 new e2e tests
• Example files in examples/convention-policies/
• Mintlify docs, README, and CHANGELOG updated

Test plan

• bun run test:run — 853 unit tests pass
• bun run test:e2e — 198 e2e tests pass (16 new convention tests)
• bun run lint — no errors
• bun run tsc --noEmit — no type errors
• bun run build — builds successfully
• CI green

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Convention-based policy auto-discovery: Automatically discovers and loads policy files from .failproofai/policies/ directories at project and user scope without requiring configuration or flags.

• Documentation

  • Added comprehensive guides, configuration examples, and sample policies for convention-based policy setup and usage.

[coderabbitai]

Warning

Rate limit exceeded

@NiveditJain has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 45 minutes and 29 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 45 minutes and 29 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d6c4f3ec-add8-4e51-ae20-6a9a3805e96b

📥 Commits

Reviewing files that changed from the base of the PR and between fe42e16 and 4e33d7f.

📒 Files selected for processing (4)

• CHANGELOG.md
• README.md
• src/hooks/custom-hooks-loader.ts
• src/hooks/manager.ts

📝 Walkthrough

Walkthrough

This PR adds convention-based policy auto-discovery, enabling automatic loading of *policies.{js,mjs,ts} files from .failproofai/policies/ directories at project and user levels without requiring configuration or CLI flags.

Changes

Cohort / File(s)	Summary
Documentation & Help CHANGELOG.md, README.md, bin/failproofai.mjs, docs/configuration.mdx, docs/custom-policies.mdx	Added feature documentation for convention-based policy discovery from .failproofai/policies/ directories (project and user scopes), file naming requirements (*policies.{js,mjs,ts}), ordering semantics, and combined load order with explicit policies. Updated CLI help text to reference convention policies.
Test Infrastructure __tests__/e2e/helpers/fixture-env.ts, __tests__/e2e/hooks/custom-hooks.e2e.test.ts, __tests__/hooks/custom-hooks-loader.test.ts, __tests__/hooks/handler.test.ts, __tests__/hooks/manager.test.ts	Extended fixture environment with writePolicyFile() helper; added comprehensive E2E test suite for convention policy discovery and execution semantics (scopes, event matching, failure modes, file filtering); added unit tests for discoverPolicyFiles() and loadAllCustomHooks() functions; updated handler and manager tests to work with new loader API returning hooks plus convention source metadata.
Example Policies examples/convention-policies/security-policies.mjs, examples/convention-policies/workflow-policies.mjs	Added sample convention policy files demonstrating security (blocking unsafe file/command patterns) and workflow policies (pre-commit testing, audit logging), ready to copy into .failproofai/policies/.
Core Policy Loading src/hooks/custom-hooks-loader.ts	Introduced discoverPolicyFiles(dir) for filesystem scanning; added loadSingleFile() for per-file loading without registry clearing; implemented loadAllCustomHooks() to load explicit custom file then convention files from both project and user scopes; added ConventionSource and LoadAllResult types; convention hooks are tagged with __conventionSource marker.
Hook Registration & Execution src/hooks/handler.ts	Updated to call loadAllCustomHooks() and use returned convention source metadata; policy registration now prefixes policies as convention/ vs. custom/ based on source; error telemetry and handler logic now include is_convention_policy flag; added convention_policies_loaded telemetry event with per-scope file counts.
Hook Management & Discovery src/hooks/manager.ts	Extended listHooks command to discover and display convention policies from project and user directories alongside existing custom policies and builtins.

Sequence Diagram

sequenceDiagram
    participant User
    participant FileSystem as Filesystem
    participant Loader as Policy Loader
    participant Registry as Hook Registry
    participant Handler as Event Handler
    participant Policy as Loaded Policy

    User->>FileSystem: Places *policies.{js,mjs,ts} files<br/>in .failproofai/policies/
    Handler->>Loader: loadAllCustomHooks(customPath)
    Loader->>FileSystem: discoverPolicyFiles(project dir)
    FileSystem-->>Loader: [sorted filenames]
    Loader->>FileSystem: discoverPolicyFiles(user dir)
    FileSystem-->>Loader: [sorted filenames]
    
    loop For each discovered file
        Loader->>FileSystem: read & import file
        FileSystem-->>Policy: module loaded
        Policy->>Registry: customPolicies.add(hook)
        Registry-->>Loader: hook registered
        Loader->>Loader: tag hook with __conventionSource
    end
    
    Loader-->>Handler: {hooks[], conventionSources[]}
    Handler->>Registry: register all policies<br/>(custom/ & convention/)
    Handler->>Handler: emit convention_policies_loaded<br/>telemetry
    
    User->>Handler: trigger tool event
    Handler->>Registry: query policies
    Registry-->>Handler: matching policies
    Handler->>Policy: execute (custom or convention)
    Policy-->>Handler: allow/deny/instruct decision
    Handler->>Handler: log is_convention_policy<br/>in telemetry

Loading

Estimated Code Review Effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly Related PRs

• Fix custom hooks loading + ESM shim exports + project-scoped LLM config #76: Modifies the same hook-loading codepaths in src/hooks/custom-hooks-loader.ts and src/hooks/handler.ts to propagate sessionCwd and refactor loader/handler APIs.

Poem

🐰 Hop, hop, hooray! Our policies now bloom,
Convention-based discovery—no config gloom!
Drop files in .failproofai/ with care,
They'll auto-load with alphabetic flair! 🌱✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main feature: convention-based policy auto-discovery. It directly reflects the primary change in the changeset.
Description check	✅ Passed	The PR description covers the what, why, and test results comprehensively. It includes a detailed summary, test execution confirmation across all required checks, and mentions documentation updates.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch luv-88-convention-policies

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (1)

examples/convention-policies/workflow-policies.mjs (1)

35-40: Consider defensive coding for the includes check.

Line 36 has a subtle edge case: if ctx.toolName is null (not undefined), the ?? operator won't trigger and includes(null) would return false (safe), but the intent is clearer with explicit null handling.

Current code is functionally correct since includes() coerces null to "null" string for comparison. No action required, just noting for clarity.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@examples/convention-policies/workflow-policies.mjs` around lines 35 - 40, The
conditional in the async fn should defensively normalize ctx.toolName before
calling includes: ensure the value used in ["Write","Edit"].includes is coerced
to a string or explicitly nullish-fallbacked (e.g., convert ctx.toolName to a
string or use an empty-string fallback) so that the includes check is
unambiguous; update the fn's condition that references ctx.toolName and the
includes call accordingly.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@CHANGELOG.md`:
- Line 6: Update the changelog entry that currently references PR `#88` to the
correct PR number `#89`; specifically edit the line mentioning "Convention-based
policy auto-discovery: ... (`#88`)" in CHANGELOG.md and change the trailing PR
reference to "(`#89`)" so the entry matches the actual PR.

In `@README.md`:
- Around line 223-230: The fenced code block in README.md is missing a language
identifier; update the triple-backtick opening fence for the directory example
to include a language (e.g., change "```" to "```text") so the static analyzer
and renderers treat it as plain text; locate the block containing
".failproofai/policies/security-policies.mjs" and
"~/.failproofai/policies/my-policies.mjs" and add the language specifier to the
opening fence.

In `@src/hooks/custom-hooks-loader.ts`:
- Around line 184-190: The code tags convention hooks by name using
conventionHookNames, which can mis-tag explicit hooks with the same name;
instead, during convention file loading collect the actual hook object
references into a Set (e.g., conventionHookSet) rather than names
(conventionSources -> collect hook objects), then iterate allHooks and mark only
those where conventionHookSet.has(hook) by setting (hook as CustomHook & {
__conventionSource?: boolean }).__conventionSource = true; update references
from conventionHookNames to the new Set to avoid name collisions.

In `@src/hooks/manager.ts`:
- Around line 670-679: Replace the fragile file.split("/") filename extraction
with Node's cross-platform basename: import basename from "node:path" alongside
the existing resolve/dirname imports, then change all occurrences where filename
is derived (currently const filename = file.split("/").pop() ?? file;) to use
const filename = basename(file); ensure this replacement is applied in the
success, failure and catch blocks around the hooks handling (the blocks
referencing filename and hooks.map(...)) so Windows backslashes are handled
correctly.

---

Nitpick comments:
In `@examples/convention-policies/workflow-policies.mjs`:
- Around line 35-40: The conditional in the async fn should defensively
normalize ctx.toolName before calling includes: ensure the value used in
["Write","Edit"].includes is coerced to a string or explicitly
nullish-fallbacked (e.g., convert ctx.toolName to a string or use an
empty-string fallback) so that the includes check is unambiguous; update the
fn's condition that references ctx.toolName and the includes call accordingly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a19f4bc8-d844-4359-a3d3-59ed7aa118ef

📥 Commits

Reviewing files that changed from the base of the PR and between e78f028 and fe42e16.

📒 Files selected for processing (15)

• CHANGELOG.md
• README.md
• __tests__/e2e/helpers/fixture-env.ts
• __tests__/e2e/hooks/custom-hooks.e2e.test.ts
• __tests__/hooks/custom-hooks-loader.test.ts
• __tests__/hooks/handler.test.ts
• __tests__/hooks/manager.test.ts
• bin/failproofai.mjs
• docs/configuration.mdx
• docs/custom-policies.mdx
• examples/convention-policies/security-policies.mjs
• examples/convention-policies/workflow-policies.mjs
• src/hooks/custom-hooks-loader.ts
• src/hooks/handler.ts
• src/hooks/manager.ts

[NiveditJain]

Superseded by #91 (combined PR with #90)