feat: apply identify, ignore and ranking-threshold client-side, add s… - #89
Open
mscasso-scanoss wants to merge 3 commits into
Open
feat: apply identify, ignore and ranking-threshold client-side, add s…#89mscasso-scanoss wants to merge 3 commits into
mscasso-scanoss wants to merge 3 commits into
Conversation
…kip-headers Migrates four scanoss.py scan options. The three result-side ones were server-side there; here they are applied to the candidate matches the v3 batch scanner returns per file, so no scan settings are sent to the server. postprocess/bomselect.go settles a file's candidate list in one pass: bom.ignore drops what the user dismissed, the ranking threshold drops what the scanner ranked too weakly, and bom.identify promotes what the user declares is present so it leads. One rule decides every interaction between them — a match an identify rule claims is never dropped — mirroring how bom.include already protected a file from bom.remove. It diverges from the engine, which applies the threshold last and lets it overwrite the identified flag; reproducing that would discard a component the user stated was present over a rule naming no component. The identify verdict lands on sbom.FileEvidence.Identified per matched file, because that is the granularity a path-scoped rule works at, with sbom.Component.Identified summarising it. CycloneDX carries the summary as a scanoss:identified property; SPDX can represent neither. Both rules match a component's canonical PURL, not its aliases: one project mined from several registries yields entries that each carry the others' PURLs, so matching aliases made one rule claim every candidate at once. wfp/headerfilter.go ports scanoss.py's HeaderFilter — the leading licence header, documentation comments and imports are dropped from a file's fingerprint, with a start_line= marker naming the offset. Offsets agree with the reference implementation on 6527 of 6531 real source files; the four that differ each hold a form feed above the first line of code, where Python's str.splitlines() counts a line the WFP's own numbering does not. All four options also read from scanoss.json, where the file overrides the flag. That is the reverse of this CLI's usual precedence and is deliberate: it is what scanoss.py does, and is documented in CLIENT_HELP.md. Fixes two pre-existing gaps found while wiring this up: `results <id>` applied no BOM rules at all, so a resumed scan reported components the project's settings dismissed; and bom.remove's protection honoured only bom.include, not bom.identify. Verified against the live API on west-1.2.0, time-1.9 and librealsense-master. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A licence header is boilerplate shared by every file that carries it, so fingerprinting it makes unrelated files look alike to the matcher. Filtering it is the better default, and opting out is --skip-headers=false. This changes the WFP every scan produces: a fingerprint taken with the filter on does not match one taken with it off. `scan wfp` still rejects the flag either way, since it is handed a WFP that is already assembled — only an explicit flag is rejected, so the new default does not make that command unusable. The default lives on the CLI flag alone. pkg/wfp and pkg/scanoss stay neutral: passing no option still means no filtering, so a library caller keeps whatever behaviour it asked for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
COMPONENT_DEFAULT_RANK is the engine's sentinel for a component it has no ranking information about, and its own "accept everything" bound is that value plus one — so an unranked component passes it. Filtering on rank was reading 999 as a very bad rank instead, and since it exceeds every threshold in range, those components were dropped under every setting rather than under none. Found by scanning the commissioning projects: rdkb-2024q4 returns 23 of its 552 components at rank 999, and they account for 2027 matched files that --ranking-threshold 1 discarded for missing data rather than for explaining a match poorly. Rank 0 was already exempt for the same reason — the field is omitted when empty, so an absent rank is indistinguishable from a zero one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…kip-headers
Migrates four scanoss.py scan options. The three result-side ones were server-side there; here they are applied to the candidate matches the v3 batch scanner returns per file, so no scan settings are sent to the server.
postprocess/bomselect.go settles a file's candidate list in one pass: bom.ignore drops what the user dismissed, the ranking threshold drops what the scanner ranked too weakly, and bom.identify promotes what the user declares is present so it leads. One rule decides every interaction between them — a match an identify rule claims is never dropped — mirroring how bom.include already protected a file from bom.remove. It diverges from the engine, which applies the threshold last and lets it overwrite the identified flag; reproducing that would discard a component the user stated was present over a rule naming no component.
The identify verdict lands on sbom.FileEvidence.Identified per matched file, because that is the granularity a path-scoped rule works at, with sbom.Component.Identified summarising it. CycloneDX carries the summary as a scanoss:identified property; SPDX can represent neither.
Both rules match a component's canonical PURL, not its aliases: one project mined from several registries yields entries that each carry the others' PURLs, so matching aliases made one rule claim every candidate at once.
wfp/headerfilter.go ports scanoss.py's HeaderFilter — the leading licence header, documentation comments and imports are dropped from a file's fingerprint, with a start_line= marker naming the offset. Offsets agree with the reference implementation on 6527 of 6531 real source files; the four that differ each hold a form feed above the first line of code, where Python's str.splitlines() counts a line the WFP's own numbering does not.
All four options also read from scanoss.json, where the file overrides the flag. That is the reverse of this CLI's usual precedence and is deliberate: it is what scanoss.py does, and is documented in CLIENT_HELP.md.
Fixes two pre-existing gaps found while wiring this up:
results <id>applied no BOM rules at all, so a resumed scan reported components the project's settings dismissed; and bom.remove's protection honoured only bom.include, not bom.identify.Verified against the live API on west-1.2.0, time-1.9 and librealsense-master.