Skip to content

Add a performance-benchmarking suite (per module/submodule) with result storage & regression comparison #86

Description

@kalwalt

Summary

Add a performance-benchmarking setup so we can measure each module/submodule's throughput, catch regressions, and quantify optimizations. Include a storage/comparison mechanism to track results over time.

Motivation

jsfeatNext is a real-time CV library (WebARKit) where per-frame cost matters, but we currently have no performance signal — only correctness (the 63-test parity suite). We can't tell if a refactor slowed gaussian_blur down, or by how much a future optimization helps. The #41 shared-cache change, for example, was argued on memory grounds with no throughput numbers to back it. A bench suite closes that gap.

Proposed approach

Harness: Vitest's built-in bench() (tinybench under the hood).
Already available — the repo runs Vitest 4.1.10. No new major dependency, reuses the existing config, TS setup, and the vendored jsfeat oracle. Convention: colocate *.bench.ts next to the code (or under bench/), run via a new "bench": "vitest bench" script.

Coverage: per module AND per submodule (as requested). One bench file per algorithm module, with separate cases for hot submodules/paths, e.g.:

  • imgprocgrayscale, gaussian_blur (+ convol.ts u8 vs f32 paths), resample (+ resample.ts u8 fast path vs float path), sobel/scharr_derivatives, canny, warp_perspective/affine
  • orbdescribe (+ bit_pattern_31.ts, rectify_patch.ts)
  • yape / yape06detect (+ yape_utils.ts)
  • fast_corners.detect, optical_flow_lk.track, linalg (svd/lu/cholesky/eigenVV), matmath, math (qsort/gaussian kernel), transform, motion_estimator (ransac/lmeds), the cache pool
  • Optionally: jsfeatNext vs the vendored jsfeat oracle side-by-side, to track parity of performance, not just output.

Deterministic inputs: reuse the synthetic deterministic image/matrix generators already in tests/ (fixed test patterns, seeded PRNGs) so runs are stable and comparable across machines/commits.

Storage & comparison mechanism:

  1. vitest bench --outputJson bench/results/<name>.json to emit structured results (hz, mean, p99, samples).
  2. Commit a baseline JSON snapshot; a small script (or Vitest's --compare) diffs a new run against it and flags regressions beyond a threshold (e.g. >10% slower).
  3. Later: optionally wire into CI as a non-blocking informational job (bench numbers on shared runners are noisy — report, don't gate), and/or keep a bench/HISTORY.md or append-only JSON log to chart trends across releases.

Scope / phasing (incremental, one module per PR — same as #47)

  1. Harness + bench script + shared deterministic-input helpers + one exemplar module (imgproc) end-to-end, including the JSON output + baseline compare script.
  2. Fill in remaining modules/submodules, one PR each, parity suite staying green.
  3. (Optional, later) CI integration + historical storage.

Impact

Low/Medium — additive, no runtime or public-API change; new dev tooling only. High long-term value: makes performance a tracked, first-class property before/through the 1.0 line.

Open questions

  • Storage: commit baseline JSON in-repo (simple, versioned, diff-able) vs an external store/dashboard (richer, more setup)? Leaning in-repo to start.
  • Include the jsfeat oracle as a comparison baseline in each bench, or bench jsfeatNext alone? (Oracle comparison is more insightful but ~doubles run time.)
  • Regression threshold + whether CI should ever gate on it (recommend: never gate; informational only, given runner noise).

Non-goals

  • Not micro-optimizing anything in this issue — this is measurement infrastructure only.
  • Not a browser/rendering benchmark (that's app-level; this is library-level throughput).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions