feat(scan): classify snippet matches as false positives (opt-in) - #98
Open
mscasso-scanoss wants to merge 1 commit into
Open
mscasso-scanoss wants to merge 1 commit into
mscasso-scanoss wants to merge 1 commit into
Conversation
Integrate github.com/scanoss/snippets-classifier to score each snippet
match on whether its reported OSS and local line ranges correspond. The
classifier is offline, but scoring needs the matched OSS file, which is
not local — so a new pipeline stage fetches it and annotates the result.
- pkg/scanoss: Contents.File fetches raw source by MD5 via
GET /v3/file-contents/{md5} (whole file or a line range).
- pkg/snippetmatch: an Annotator that reads each local file, fetches
every distinct OSS file once (cache by hash, bounded concurrency),
classifies, and sets FileEvidence.SnippetClassification. Non-fatal:
a match it cannot score is left unannotated.
- sbom: SnippetClassification {verdict, probability, model_version} on
FileEvidence, rendered in the raw output.
- scanpipeline: opt-in ClassifySnippets stage after enrichment, with a
"classify" progress layer.
- cmd: --classify-snippets flag (path scan, raw format only).
The classifier only proposes (threshold 0.7); it removes nothing.
snippets-classifier is a PRIVATE module: CI and release now set
GOPRIVATE and authenticate git with the SCANOSS_MODULE_TOKEN secret,
which must be created (a token with read access to the repo). Local
builds need GOPRIVATE too — documented in the Makefile.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Integrates
github.com/scanoss/snippets-classifierto score each snippet match on whether its reported OSS and local line ranges correspond — proposing wrong candidate assignments (false positives) for a human to confirm.The classifier itself is offline and embedded, but scoring a match needs three inputs: the local file, the matched OSS file, and the paired line ranges. The inventory carries the ranges and the OSS file's hash; the local file is on disk; the OSS file is fetched by hash. That fetch is the only network cost, so distinct OSS files are downloaded once and reused.
It only proposes (threshold 0.7) — it removes nothing.
Changes
pkg/scanoss/filecontents.go—Contents.File(ctx, md5, start, end)→GET /v3/file-contents/{md5}(whole file or a line range).pkg/snippetmatch/—Annotator: reads each local file, fetches every distinct OSS file once (cache by hash, bounded concurrency), classifies in parallel, setsFileEvidence.SnippetClassification. Non-fatal: a match it cannot score (no local file / no OSS content) is left unannotated and logged.pkg/sbom—SnippetClassification {verdict, probability, model_version}onFileEvidence, rendered inrawoutput.pkg/scanpipeline— opt-inClassifySnippetsstage after enrichment; newclassifyprogress layer.cmd/scan.go—--classify-snippetsflag (path scan only,rawformat only; warns if used with another format).Usage
Each snippet match in the raw output gains:
snippets-classifieris a private repo. CI and release now setGOPRIVATE=github.com/scanoss/*and authenticate git with aSCANOSS_MODULE_TOKENsecret. This secret must be created (a PAT or GitHub App token with read access toscanoss/snippets-classifier) or CI will fail to download the module. Local builds needGOPRIVATEtoo — documented in the Makefile.Testing
go build ./...,go vet ./...,gofmtclean; fullgo test ./...and-raceon the concurrent packages all pass.snippetmatch(real vs false-positive verdicts with the classifier's own fixtures, dedup-by-hash, non-fatal skips, non-snippet evidence ignored) andContents.File(whole-file, line range, 404).🤖 Generated with Claude Code