Skip to content

test: prune low-signal unit tests and add E2E-first test rules - #267

Merged
teslamint merged 9 commits into
mainfrom
claude/sharp-newton-1jlhwv
Sep 25, 2026
Merged

teslamint merged 9 commits into
mainfrom
claude/sharp-newton-1jlhwv

Conversation

@teslamint

@teslamint teslamint commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Purpose

Delete the unit tests that would not catch a real bug the E2E suite misses. Add test-writing rules to AGENTS.md so agents stop writing that kind of test.

Key changes

  • AGENTS.md: new ### Test-writing rules subsection under ## Test:
    • Never write unit tests after writing the code.
    • Prefer E2E tests as the only testing mechanism, and end each one with a verifiable, repeatable artifact.
    • Before testing a system in isolation, write down its failure modes first.
  • tests/: 3 files and about 440 test functions removed (79 files changed, −3977 lines). A test was deleted if any of these held:
    1. Redundant with E2E: a tests/test_e2e_*.py test already asserts the same behavior on the same code path. We read the E2E assertion itself; coverage alone did not count.
    2. Low-signal: the test only checks constants or defaults, return types or key existence, mock call wiring, --help text, or "does not raise". Also removed: assertions that can never fail, such as len==0 or ..., if found: guards, and a single digit that dates always contain.
    3. Duplicate: another kept test asserts the same branch.
  • Kept on purpose:
    • Tests for code the E2E suite does not execute. E2E line coverage was about 30%, measured with pytest tests/test_e2e_*.py --cov.
    • Regression tests.
    • Migrations.
    • Data-loss, security and redaction guards.
    • Worktree safety.
    • Ranking math.
    • Hypothesis property tests.
  • No src/, conftest.py or E2E files changed.

Note: the deletion of tests/test_config.py (two DEFAULT_CONFIG constant checks) landed in the AGENTS.md commit 4812540, because a concurrent git rm was already staged when that commit was made.

Test evidence

uv run ruff check tests/               → All checks passed!
uv run ruff format --check tests/      → 141 files already formatted
uv run pytest -q                       → 1 failed, 2168 passed, 1 skipped   (before: 1 failed, 2551 passed, 1 skipped)
uv run pytest -q tests/test_e2e_*.py   → 87 passed

The one local failure is tests/test_pdi_optimizer.py::TestEstimateTokens::test_encoding_initialized_at_module_level. It fails the same way before this PR's changes and only in the local environment. On this PR's head, CI test (3.12) and test (3.13) both pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PhNKoUPD9knX59FoMD9dX8

Summary by CodeRabbit

  • Tests
    • Updated coverage for date-bounded search, decision outcomes and ranking, embedding behavior, and other workflows.
    • Removed numerous existing test cases across search, session, repository, and decision features; no changes to product behavior are described.
  • Documentation
    • Added guidance on test-first development, end-to-end testing, and documenting failure modes when testing in isolation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhNKoUPD9knX59FoMD9dX8
Removes tests that duplicate E2E assertions, only verify mock wiring,
or repeat another kept test on the same branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhNKoUPD9knX59FoMD9dX8
…suites

Removes tests that duplicate E2E assertions, only verify mock wiring
or constants, or repeat another kept test on the same branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhNKoUPD9knX59FoMD9dX8
…o suites

Removes tests that duplicate E2E assertions, only check constants,
key existence, or mock wiring, or repeat another kept test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhNKoUPD9knX59FoMD9dX8
…compact suites

Removes tests that duplicate E2E assertions, only check constants,
return types, key existence, or mock wiring, or repeat another kept test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhNKoUPD9knX59FoMD9dX8
…arch suites

Removes tests that duplicate E2E assertions, only check constants,
return types, key existence, or mock wiring, or repeat another kept test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhNKoUPD9knX59FoMD9dX8
… suites

Removes tests that duplicate E2E assertions, only check constants,
return types, key existence, or mock wiring, or repeat another kept test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhNKoUPD9knX59FoMD9dX8
…ort suites

Removes tests that duplicate E2E assertions, only check constants,
return types, key existence, or mock wiring, or repeat another kept test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhNKoUPD9knX59FoMD9dX8
… suites

Removes tests that duplicate E2E assertions, only check return types,
key existence, or mock wiring, or repeat another kept test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhNKoUPD9knX59FoMD9dX8
@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: teslamint/entirecontext/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 07dd3954-54d9-4ac8-94c4-d6b59a742bfa

📥 Commits

Reviewing files that changed from the base of the PR and between 8c478fa and 24af9c2.

📒 Files selected for processing (80)
  • AGENTS.md
  • tests/test_aar.py
  • tests/test_activation.py
  • tests/test_agent_graph.py
  • tests/test_archaeology.py
  • tests/test_archaeology_cli.py
  • tests/test_archaeology_integration.py
  • tests/test_archaeology_streaming.py
  • tests/test_assessment_relationships.py
  • tests/test_ast_index.py
  • tests/test_async_worker.py
  • tests/test_auto_assess.py
  • tests/test_auto_sync.py
  • tests/test_blame_decisions.py
  • tests/test_checkpoint.py
  • tests/test_checkpoint_create.py
  • tests/test_cli.py
  • tests/test_compact.py
  • tests/test_config.py
  • tests/test_consolidation.py
  • tests/test_content_filter.py
  • tests/test_context.py
  • tests/test_core.py
  • tests/test_cross_repo.py
  • tests/test_cross_repo_expanded.py
  • tests/test_cross_repo_futures.py
  • tests/test_dashboard.py
  • tests/test_db_connection_guard.py
  • tests/test_db_schema.py
  • tests/test_decision_alternatives.py
  • tests/test_decision_candidates_batch.py
  • tests/test_decision_embedding.py
  • tests/test_decision_extraction.py
  • tests/test_decision_hooks.py
  • tests/test_decision_verify.py
  • tests/test_decisions_cli.py
  • tests/test_decisions_core.py
  • tests/test_embedding.py
  • tests/test_event.py
  • tests/test_event_cmds.py
  • tests/test_experiment_block.py
  • tests/test_export.py
  • tests/test_extraction_archaeology_seam.py
  • tests/test_futures_report.py
  • tests/test_git_isolation.py
  • tests/test_handler.py
  • tests/test_hook_cmds.py
  • tests/test_hooks.py
  • tests/test_hooks_performance.py
  • tests/test_hybrid_search.py
  • tests/test_import_aline.py
  • tests/test_import_cmds.py
  • tests/test_index_cmds.py
  • tests/test_indexing.py
  • tests/test_knowledge_graph.py
  • tests/test_llm.py
  • tests/test_mcp.py
  • tests/test_mcp_cmds.py
  • tests/test_mcp_input_normalization.py
  • tests/test_migration_v016.py
  • tests/test_outcome_inference.py
  • tests/test_pdi_handler.py
  • tests/test_pdi_optimizer.py
  • tests/test_post_commit_hook.py
  • tests/test_project_cmds.py
  • tests/test_ranking_snapshot_backpatch.py
  • tests/test_ranking_snapshot_purge.py
  • tests/test_repo_cmds.py
  • tests/test_resolve_and_helpers.py
  • tests/test_search_cmds.py
  • tests/test_search_redaction.py
  • tests/test_session_cmds.py
  • tests/test_signal_b.py
  • tests/test_sync.py
  • tests/test_sync_engine.py
  • tests/test_tidy_pr.py
  • tests/test_token_savings_experiment.py
  • tests/test_tql.py
  • tests/test_transaction_helper.py
  • tests/test_verdict_accuracy.py
💤 Files with no reviewable changes (73)
  • tests/test_index_cmds.py
  • tests/test_aar.py
  • tests/test_db_schema.py
  • tests/test_verdict_accuracy.py
  • tests/test_signal_b.py
  • tests/test_import_aline.py
  • tests/test_indexing.py
  • tests/test_transaction_helper.py
  • tests/test_token_savings_experiment.py
  • tests/test_sync_engine.py
  • tests/test_handler.py
  • tests/test_resolve_and_helpers.py
  • tests/test_checkpoint_create.py
  • tests/test_decision_candidates_batch.py
  • tests/test_async_worker.py
  • tests/test_decision_verify.py
  • tests/test_import_cmds.py
  • tests/test_archaeology.py
  • tests/test_pdi_optimizer.py
  • tests/test_outcome_inference.py
  • tests/test_auto_assess.py
  • tests/test_ranking_snapshot_purge.py
  • tests/test_ranking_snapshot_backpatch.py
  • tests/test_git_isolation.py
  • tests/test_checkpoint.py
  • tests/test_archaeology_streaming.py
  • tests/test_consolidation.py
  • tests/test_config.py
  • tests/test_search_cmds.py
  • tests/test_mcp_cmds.py
  • tests/test_session_cmds.py
  • tests/test_archaeology_integration.py
  • tests/test_archaeology_cli.py
  • tests/test_hooks_performance.py
  • tests/test_blame_decisions.py
  • tests/test_event_cmds.py
  • tests/test_experiment_block.py
  • tests/test_auto_sync.py
  • tests/test_decisions_cli.py
  • tests/test_futures_report.py
  • tests/test_post_commit_hook.py
  • tests/test_decision_embedding.py
  • tests/test_assessment_relationships.py
  • tests/test_repo_cmds.py
  • tests/test_db_connection_guard.py
  • tests/test_hook_cmds.py
  • tests/test_tidy_pr.py
  • tests/test_cross_repo_futures.py
  • tests/test_mcp_input_normalization.py
  • tests/test_ast_index.py
  • tests/test_agent_graph.py
  • tests/test_cli.py
  • tests/test_migration_v016.py
  • tests/test_activation.py
  • tests/test_hybrid_search.py
  • tests/test_dashboard.py
  • tests/test_export.py
  • tests/test_pdi_handler.py
  • tests/test_llm.py
  • tests/test_decision_extraction.py
  • tests/test_search_redaction.py
  • tests/test_project_cmds.py
  • tests/test_compact.py
  • tests/test_event.py
  • tests/test_embedding.py
  • tests/test_tql.py
  • tests/test_hooks.py
  • tests/test_cross_repo.py
  • tests/test_extraction_archaeology_seam.py
  • tests/test_decision_alternatives.py
  • tests/test_cross_repo_expanded.py
  • tests/test_mcp.py
  • tests/test_content_filter.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds test-writing guidance and changes tests across many areas. Most changes remove existing test cases. Some tests are added or revised for decision behavior, hook workflows, embedding behavior, search date handling, and tidy-PR output. No production-code changes are described.

Changes

Test suite updates

Layer / File(s) Summary
Repository state and persistence tests
AGENTS.md, tests/test_checkpoint*.py, tests/test_context.py, tests/test_core.py, tests/test_db_*.py, tests/test_event*.py, tests/test_resolve_and_helpers.py, tests/test_session_cmds.py, tests/test_transaction_helper.py
AGENTS.md adds test-writing guidance. Tests for checkpoint, session, database, event, transaction, and helper behavior are removed or narrowed.
Search, indexing, and temporal filters
tests/test_ast_index.py, tests/test_cross_repo*.py, tests/test_embedding.py, tests/test_hybrid_search.py, tests/test_indexing.py, tests/test_search*.py, tests/test_tql.py
Search and indexing tests are removed or revised. Added search-command tests check --until conversion, fallback behavior, and rejection of reversed date ranges.
Decision extraction, outcomes, and ranking
tests/test_assessment_relationships.py, tests/test_decision*.py, tests/test_decisions*.py, tests/test_outcome_inference.py, tests/test_ranking_snapshot*.py, tests/test_verdict_accuracy.py
Decision tests are removed or revised across extraction, alternatives, outcome recording, ranking, supersession, and verification. Added tests cover outcome types, score composition, cycle detection, and transaction rollback.
Agent, hook, and worker workflows
tests/test_activation.py, tests/test_agent_graph.py, tests/test_async_worker.py, tests/test_auto_sync.py, tests/test_decision_hooks.py, tests/test_handler.py, tests/test_hook*.py, tests/test_post_commit_hook.py
Tests for agent, hook, worker, and sync workflows are removed or revised. Added hook tests cover terminal-successor substitution, extraction filtering, null-turn inference, repository-path forwarding, and fallback-file cleanup.
CLI and feature integration tests
tests/test_aar.py, tests/test_archaeology*.py, tests/test_cli.py, tests/test_compact.py, tests/test_dashboard.py, tests/test_export.py, tests/test_futures_report.py, tests/test_knowledge_graph.py, tests/test_mcp*.py, tests/test_project_cmds.py, tests/test_tidy_pr.py
Tests for CLI and feature integrations are removed or revised. Tidy-PR tests add checks for included suggestion text and YAML frontmatter.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 24af9

No actionable merge-blocking issue is established. The changes are mergeable after normal checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 29.31% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 58 functions across 6 files. (1 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes both main changes: pruning low-signal unit tests and adding E2E-first test rules.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 29.31% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 58 functions across 6 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sonarqubecloud

Copy link
Copy Markdown

@teslamint
teslamint marked this pull request as ready for review September 25, 2026 23:48
@teslamint teslamint self-assigned this Sep 25, 2026
@teslamint teslamint added the enhancement New feature or request label Sep 25, 2026
@teslamint
teslamint merged commit 2d85ece into main Sep 25, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants