perf(bench): add reproducible large-monorepo benchmark and scale report

[KumamuKuma]

Summary

This PR adds a reproducible benchmark for the non-LLM stages of /understand:
file scanning, import extraction, batch planning, and Tree-sitter structure
extraction.

It gives contributors a shared way to measure large repositories and find
bottlenecks using repeatable evidence. It does not call an LLM or write
analysis data into the repository being tested.

Each run produces matching JSON and Markdown reports with repository scale,
timing, CPU and memory use, warnings, failures, and input/output digests.
Private paths, hostnames, and Git remotes are not included.

Usage

corepack pnpm benchmark:large-repo /absolute/path/to/repository \
  --label public-repository-name \
  --output /absolute/path/to/results/report.json

The output must be outside the tested repository. Temporary files are cleaned
up by default.

What this adds

• Deterministic file selection, ordering, batching, and content digests.
• Isolated benchmark stages with bounded concurrency and failure reporting.
• Paired JSON/Markdown reports validated by schema version 1.0.0.
• Cross-platform instructions and a pinned TensorFlow reproduction recipe.

Related work

• Refs How to make /understand run faster ? #76 for /understand performance work.
• Refs Investigation: evaluate Understand-Anything for onboarding onto large codebases #483 for large-codebase evaluation.
• Follow-up to the scale discussion around fix(dashboard): move force layout off the main thread #576.

Limitations

• This is not an end-to-end /understand benchmark. It does not run an LLM,
  count tokens, estimate cost, build the final knowledge graph, or open the
  dashboard.
• estimatedAgentInputBytes measures bytes, not tokens or cost.
• TensorFlow is currently a pinned recipe, not a completed measurement.
• This PR does not claim verified support for one-million-line repositories.
• Timing and memory should only be compared across same-machine runs with the
  same concurrency and matching digests.

Current plan

1. Land the reproducible benchmark and report format.
2. Collect at least three same-digest community runs, including the pinned
   TensorFlow revision and other monorepos.
3. Compare repeated runs on the same machine.
4. Use the results to choose and verify the next optimization.
5. Benchmark LLM, token, and cost behavior separately in future work.

Testing

• Install, lint, and core/skill/viewer builds passed.
• Core tests: 934 passed.
• Root Vitest suite: 380 passed, 7 platform-specific skips.
• Python helpers: 95 passed, 1 platform-specific skip.
• Benchmark/schema tests: 70 passed, 1 platform-specific skip.
• A real CLI smoke test produced schema-valid paired reports, leaked no private
  paths, and left the tested repository clean.

Versioning

N/A — this PR adds internal benchmark tooling, tests, and documentation.

[thejesh23]

@KumamuKuma Can you please share the sample report to manual review, it would be easier to visualize the benchmarking result expectation through this PR. If you already have one for tensor-flow you can share that as well

[KumamuKuma]

@thejesh23 Thanks for the suggestion. I added full-repository sample reports for manual review:

• Markdown report
• JSON report

The sample covers this repository itself: 459 files and 127,155 lines. It shows repository scale, stage timing, CPU and memory usage, warnings, and integrity information for the deterministic non-LLM stages. The JSON validates against schema 1.0.0, and its pair ID matches the Markdown report.

I have not generated a TensorFlow report locally because its size requires more time and resources than I can reliably provide in my current setup. The pinned TensorFlow reproduction recipe remains available in this PR for community testing.

While preparing the sample, I noticed a small report-classification issue, so I added a follow-up commit to document the expected behavior, correct the issue, strengthen result validation, and include the final sample reports.

[KumamuKuma]

@thejesh23 Thanks again for reviewing this.
My next plan is to collect more same-digest benchmark runs from large repositories, especially the pinned TensorFlow revision. I would then compare repeated runs made on the same machine with the same concurrency, identify which stage is consistently the main bottleneck, and use that evidence to choose and verify the next optimization. LLM, token, and cost measurements would remain a separate follow-up.
If you have time, would you be willing to help run the pinned TensorFlow benchmark and share the paired Markdown and JSON reports? The tool records the repository commit, environment, digests, timing, memory usage, and integrity results, so the reports can be compared reliably. A run from TensorFlow—or another large monorepo—would be very helpful.

[thejesh23]

@KumamuKuma Sure I can and I will . Will do it after I'm back from vacation next week.