Skip to content

feat(scan): classify snippet matches as false positives (opt-in) - #98

Open
mscasso-scanoss wants to merge 1 commit into
mainfrom
feat/snippet-classifier
Open

mscasso-scanoss wants to merge 1 commit into
mainfrom
feat/snippet-classifier

Conversation

@mscasso-scanoss

Copy link
Copy Markdown
Contributor

What

Integrates github.com/scanoss/snippets-classifier to score each snippet match on whether its reported OSS and local line ranges correspond — proposing wrong candidate assignments (false positives) for a human to confirm.

The classifier itself is offline and embedded, but scoring a match needs three inputs: the local file, the matched OSS file, and the paired line ranges. The inventory carries the ranges and the OSS file's hash; the local file is on disk; the OSS file is fetched by hash. That fetch is the only network cost, so distinct OSS files are downloaded once and reused.

It only proposes (threshold 0.7) — it removes nothing.

Changes

  • pkg/scanoss/filecontents.go — Contents.File(ctx, md5, start, end) → GET /v3/file-contents/{md5} (whole file or a line range).
  • pkg/snippetmatch/ — Annotator: reads each local file, fetches every distinct OSS file once (cache by hash, bounded concurrency), classifies in parallel, sets FileEvidence.SnippetClassification. Non-fatal: a match it cannot score (no local file / no OSS content) is left unannotated and logged.
  • pkg/sbom — SnippetClassification {verdict, probability, model_version} on FileEvidence, rendered in raw output.
  • pkg/scanpipeline — opt-in ClassifySnippets stage after enrichment; new classify progress layer.
  • cmd/scan.go — --classify-snippets flag (path scan only, raw format only; warns if used with another format).

Usage

scanoss-cli scan ./src --classify-snippets

Each snippet match in the raw output gains:

"snippet_classification": { "verdict": "false_positive", "probability": 0.91, "model_version": "lr-…" }

⚠️ Required before merge: private-module auth

snippets-classifier is a private repo. CI and release now set GOPRIVATE=github.com/scanoss/* and authenticate git with a SCANOSS_MODULE_TOKEN secret. This secret must be created (a PAT or GitHub App token with read access to scanoss/snippets-classifier) or CI will fail to download the module. Local builds need GOPRIVATE too — documented in the Makefile.

Testing

  • go build ./..., go vet ./..., gofmt clean; full go test ./... and -race on the concurrent packages all pass.
  • New unit tests: snippetmatch (real vs false-positive verdicts with the classifier's own fixtures, dedup-by-hash, non-fatal skips, non-snippet evidence ignored) and Contents.File (whole-file, line range, 404).
  • Not exercised end-to-end here (needs a live API key + a scan producing snippet matches).

🤖 Generated with Claude Code

Integrate github.com/scanoss/snippets-classifier to score each snippet
match on whether its reported OSS and local line ranges correspond. The
classifier is offline, but scoring needs the matched OSS file, which is
not local — so a new pipeline stage fetches it and annotates the result.

- pkg/scanoss: Contents.File fetches raw source by MD5 via
  GET /v3/file-contents/{md5} (whole file or a line range).
- pkg/snippetmatch: an Annotator that reads each local file, fetches
  every distinct OSS file once (cache by hash, bounded concurrency),
  classifies, and sets FileEvidence.SnippetClassification. Non-fatal:
  a match it cannot score is left unannotated.
- sbom: SnippetClassification {verdict, probability, model_version} on
  FileEvidence, rendered in the raw output.
- scanpipeline: opt-in ClassifySnippets stage after enrichment, with a
  "classify" progress layer.
- cmd: --classify-snippets flag (path scan, raw format only).

The classifier only proposes (threshold 0.7); it removes nothing.

snippets-classifier is a PRIVATE module: CI and release now set
GOPRIVATE and authenticate git with the SCANOSS_MODULE_TOKEN secret,
which must be created (a token with read access to the repo). Local
builds need GOPRIVATE too — documented in the Makefile.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant