Skip to content

Strengthen tests beyond parity: property/invariant tests, edge cases, third-party ground truth #87

Description

@kalwalt

Summary

The test suite is currently parity-only: all 60 cases compare jsfeatNext against the vendored original-jsfeat oracle on (mostly) a single fixed input each. That's an excellent refactor safety net but not sufficient correctness testing for a 1.0 library. Complement it with oracle-independent property tests, edge/boundary cases, and a few third-party ground-truth values.

Motivation — the three ceilings of parity-only testing

  1. Parity can never catch a bug present in both libraries. The oracle is original jsfeat, so these tests prove "identical behavior to jsfeat," not "correct." They inherit jsfeat's bugs by construction. (We saw both sides: the jsfeatNext-only affine2d bug Bug: ransac/lmeds with affine2d kernel throws TypeError (missing motion_model.check_subset) #51 was caught by parity; jsfeat's own broken hough_transform never could be.) For 1.0, "bug-for-bug identical" ≠ "correct."
  2. Almost everything runs on one fixed input. imgproc tests a single 64×48 synthetic pattern, one input per function — no boundary conditions (1×1, empty/uniform images, min/max intensities, odd-vs-even dims, tiny kernels, singular/degenerate matrices), which is exactly where CV code breaks.
  3. No oracle-independent checks. No property/invariant tests — the kind true regardless of jsfeat, and the only kind that will work where no oracle exists (future haar/bbf, Port haar module (Haar cascade object/face detection) #43/Port bbf module (BBF face detection) #44) or where we intentionally diverge (e.g. the transform matrix_t signature).

Proposed additions (keep the parity suite; layer on top)

A. Property / invariant tests (oracle-independent correctness)

  • matmath/transform: invert(invert(M)) ≈ M; M · invert(M) ≈ I; warp-then-unwarp round-trips to ~identity
  • linalg: SVD reconstruction U · diag(W) · Vᵀ ≈ A; lu_solve/cholesky_solve residual ‖A·x − b‖ ≈ 0; eigenVV eigen-equation A·v ≈ λ·v
  • imgproc: output-range guarantees (grayscale/blur ∈ [0,255]); box_blur/gaussian_blur of a uniform image returns the same uniform value; integral-image corner sums equal brute-force region sums
  • detectors: idempotence/structural invariants (e.g. detected corner counts monotic in threshold)

B. Edge / boundary inputs (add as extra cases to existing module tests)

  • 1×1 and Nx1/1xN images; empty and uniform images; extreme intensities (all 0, all 255); odd-vs-even dimensions; minimum kernel sizes; singular / ill-conditioned matrices for the solvers.

C. A few third-party ground-truth values

  • Hand-computed or OpenCV-sourced expected outputs for a handful of core functions (grayscale luma, a 3×3 SVD, a known homography), so we are not 100% circular with jsfeat.

D. Coverage gaps

  • data_type has no dedicated test (only module lacking one). Confirm yape (the non-yape06 detector) has real coverage, not just an import.

Scope / phasing

Additive and incremental — one module (or one category) per PR, existing parity suite staying green. Suggest starting with the linalg invariants (highest bug-catching value, purely mathematical) as the exemplar.

Impact

Low/Medium — test-only, no runtime/API change. High value: real correctness signal + safety for the modules where jsfeat can't be an oracle. Pre-1.0 quality gate (relates to #85, #86).

Non-goals

  • Not removing or replacing the parity suite — it stays as the refactor safety net.
  • Not a coverage-percentage target for its own sake.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions