Skip to content

evaluation.md

Latest commit

 

History

History
49 lines (31 loc) · 3.34 KB

evaluation.md

File metadata and controls

49 lines (31 loc) · 3.34 KB
Raw

Evaluation

scalps is evaluated with focused fixtures for extraction and scoring behavior plus a real-project ranking check from scalpel-editor. These checks are regression evidence, not a claim that the tool solves general natural-language code search.

What the automated tests cover

The test suite covers definition identity, exact written ranges, namespaces, overloads, templates, inline header definitions, operator names, macro boundaries, compilation-database validation, complete index replacement, transaction rollback, candidate retrieval sources, every score contribution, sibling ranking, and deterministic ordering.

The compact scalpel-editor ranking fixture contains 28 copied definition records representing the three target definitions and their main competitors. It requires:

  • Editor::SetWrapMode ranks first for set line wrap mode.
  • Editor::SearchInTarget ranks in the top three for find text in target.
  • ScintillaBase::AutoCShow ranks in the top five for show an autocomplete list.

The fixture includes attached documentation and competitors such as SetMarginSensitiveN, GetWrapMode, SetPrintWrapMode, Document::FindText, AutoComplete::Show, and AutoComplete constructors. It keeps the ranking defect reproducible without requiring a separate scalpel-editor checkout during ordinary tests.

Run all automated checks with:

ctest --test-dir build --output-on-failure

Full scalpel-editor index

The full evaluation project contains 111 accepted compilation commands. With schema version 7, extractor version 5, and ranking configuration version 4, indexing produced:

  • 111 successful translation units and no failed translation units.
  • 4,434 merged definitions.
  • No skipped eligible project definitions.
  • No merge conflicts.
  • 346 unique fully macro-generated exclusions, primarily Catch test definitions.

The motivating query results in that full index were:

Query Expected definition Rank
set line wrap mode Editor::SetWrapMode 1
find text in target Editor::SearchInTarget 1
show an autocomplete list ScintillaBase::AutoCShow 1

These ranks are tied to the recorded extractor and ranking versions. A future scoring or extraction change should rerun the full index and update this document only after the corresponding fixture and focused tests pass.

Evaluating a ranking defect

When a useful query ranks poorly, rerun it with --verbose and record the query, expected qualified definition, actual rank, index versions, and score explanations for the leading candidates. Inspect whether the failure comes from candidate generation, identifier analysis, operation classification, structural scoring, lexical field balance, or tie-breaking before changing weights.

Prefer structural corrections that improve a class of searches over special vocabulary. Add a narrow normalization rule only when the structural and field weighting are sound and a concrete naming convention still prevents the intended match. Avoid modifying the searched project's source or comments merely to influence the benchmark.

Capture a compact fixture containing the intended definition and the competitors that demonstrate the imbalance. Require a useful rank bound rather than an unnecessarily exact order for unrelated candidates, and keep the existing sibling and deterministic-order tests passing.