Skip to content

update report adding license information - #35

Merged
mscasso-scanoss merged 7 commits into
mainfrom
mscasso/feat/add-urlhash-and-lincese-to-report-SP4189
Jun 5, 2026
Merged

update report adding license information#35
mscasso-scanoss merged 7 commits into
mainfrom
mscasso/feat/add-urlhash-and-lincese-to-report-SP4189

Conversation

@mscasso-scanoss

@mscasso-scanoss mscasso-scanoss commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • New Features

    • Responses now include per-version release date, URL hash, and license; added snapshot export/restore tooling and a CSV import path with a 13-column schema and optional rank override.
  • Bug Fixes / Behavior

    • Importer validates inputs, skips short rows, defaults invalid/empty rank to 0, supports rank overrides, simplifies point IDs, and removes legacy payload fields.
  • Documentation

    • Added CSV format, import/restore procedures, examples, and snapshot usage notes.
  • Chores

    • Ignore local vector DB data directory.

@mscasso-scanoss mscasso-scanoss self-assigned this Jun 3, 2026
@coderabbitai

coderabbitai Bot commented Jun 3, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 825b5e4b-b0cb-42f4-ab02-6cb13e4a5130

📥 Commits

Reviewing files that changed from the base of the PR and between c9fac77 and 896b7a0.

📒 Files selected for processing (3)
  • CHANGELOG.md
  • README.md
  • scripts/README.md
✅ Files skipped from review due to trivial changes (2)
  • CHANGELOG.md
  • README.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • scripts/README.md

📝 Walkthrough

Walkthrough

Adds release_date, url_md5, and license through domain types, CSV importer, Qdrant extraction, and protobuf mapping; updates CSV format/docs, language constants, vector key usage, snapshot scripts, .gitignore, and bumps a module dependency.

Changes

Metadata Field Expansion

Layer / File(s) Summary
Domain entity extension
internal/domain/entities/component.go
Version and SearchResult gain ReleaseDate, URLMD5, and License fields.
Language constants & indexed extensions
internal/domain/entities/language.go
Adds internal collection-name and extSwift constants; refactors PrimaryLanguages and IndexedLangExtensions to use them.
CSV import, vector keys, and collection wiring
cmd/import/main.go
Rewrites CSV-to-payload mapping (adds url_md5, release_date, license), accepts shorter rows, derives rank from CSV with optional -top-purls override, moves language-extension parsing earlier for collection routing, narrows payload keyword indexing to purl/version, and replaces hard-coded vector keys with VectorDirs/VectorNames/VectorContents.
Qdrant payload extraction & grouping
internal/repository/scan_repository_qdrant_impl.go
Reads release_date, url_md5, and license from Qdrant payload into SearchResult; groups results by PURL and includes new fields into Version entries.
Protobuf response license mapping
internal/mapper/scan_mapper_impl.go
Builds per-version Licenses protobuf list from entity License data (adds fmt import for SPDX URL).
Documentation and changelog updates
CHANGELOG.md, README.md, scripts/README.md
CHANGELOG documents new payload/response fields and CSV reshaping; README adds a “CSV Format” section describing the 13-column headerless CSV, validation rules, routing by language_extensions, and snapshot restore guidance.
Qdrant snapshot scripts
scripts/qdrant-generate-snapshots.sh, scripts/qdrant-restore-snapshots.sh
Add scripts to snapshot and restore Qdrant collections with jq validation, endpoint override, optional snapshots dir, and per-collection upload/download semantics.
Module dependency update
go.mod
Bumps github.com/scanoss/papi version and removes commented local replace line.
.gitignore addition
.gitignore
Adds qdrant_data/ to ignore local Qdrant data.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested labels

enhancement

🐰 I munched CSV rows under moonlit code,
Added hashes, dates, and license ode,
From importer’s hop to Qdrant’s nest,
Versions now carry metadata best,
A tiny hop for me, a tidy data road. 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Title check ❓ Inconclusive The title is vague and generic. It references 'update report' and 'license information' but does not capture the substantial changes including CSV import restructuring, URL hash handling, Qdrant payload modifications, snapshot restoration tooling, and entity field extensions. Consider a more specific title that reflects the main technical changes, such as 'Add URL hash and license metadata to component versioning and reporting' or 'Restructure CSV import with URL hash, license, and snapshot restore support'.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch mscasso/feat/add-urlhash-and-lincese-to-report-SP4189

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/mapper/scan_mapper_impl.go`:
- Around line 112-123: The mapper is incorrectly assigning v.URLMD5 into
scanningv2.HFHResponse_Version.UrlHash which breaks the existing url_hash
contract; change the mapping so scanningv2.HFHResponse_Version.UrlHash is
populated with the original url_hash value (e.g., v.UrlHash or whatever field
carries the original url_hash in the domain model) and expose v.URLMD5 via a new
protobuf field (e.g., UrlMd5) on scanningv2.HFHResponse_Version; update the
append in the function that builds versions to set UrlHash: v.UrlHash and
UrlMd5: v.URLMD5 (and add the new proto field and regenerate types if not
present) while leaving repository/domain persistence of url_hash unchanged.

In `@README.md`:
- Around line 188-191: Update the README note that currently says "Rows with
fewer than 11 fields are skipped" to match the importer behavior in
cmd/import/main.go which now skips rows with fewer than 13 fields; locate the
README's "Rows with fewer than X fields are skipped" sentence and change the
count to 13 so the documentation aligns with the Importer (cmd/import/main.go)
logic and won't mislead users.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 23b54b0f-eef0-45fc-b773-9fd78e417a3a

📥 Commits

Reviewing files that changed from the base of the PR and between 5ed652c and 4f91c20.

📒 Files selected for processing (7)
  • CHANGELOG.md
  • README.md
  • cmd/import/main.go
  • go.mod
  • internal/domain/entities/component.go
  • internal/mapper/scan_mapper_impl.go
  • internal/repository/scan_repository_qdrant_impl.go

Comment thread internal/mapper/scan_mapper_impl.go
Comment thread README.md

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
internal/domain/entities/language.go (1)

83-83: ⚡ Quick win

Remove duplicate extSwift from indexed extensions.

extSwift appears twice in IndexedLangExtensions, which adds redundant processing and makes the taxonomy inconsistent. Keep it in a single category.

Proposed fix
 var IndexedLangExtensions = []string{
@@
-	"c", "h", "cpp", "cxx", "cc", "hpp", "hxx", "m", "mm", extSwift,
+	"c", "h", "cpp", "cxx", "cc", "hpp", "hxx", "m", "mm",
@@
 	"dart", "kotlin", extSwift, "gradle",

Also applies to: 91-91

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/domain/entities/language.go` at line 83, IndexedLangExtensions
contains duplicate entries of the symbol extSwift; remove the redundant extSwift
occurrence(s) from the IndexedLangExtensions slice (references to
IndexedLangExtensions in internal/domain/entities/language.go) so each extension
appears only once—search for extSwift in the IndexedLangExtensions definition
and delete the duplicate(s), leaving a single extSwift entry and keeping all
other entries unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@internal/domain/entities/language.go`:
- Line 83: IndexedLangExtensions contains duplicate entries of the symbol
extSwift; remove the redundant extSwift occurrence(s) from the
IndexedLangExtensions slice (references to IndexedLangExtensions in
internal/domain/entities/language.go) so each extension appears only once—search
for extSwift in the IndexedLangExtensions definition and delete the
duplicate(s), leaving a single extSwift entry and keeping all other entries
unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ac37a89d-7bae-4a8a-9a93-9395177bae6b

📥 Commits

Reviewing files that changed from the base of the PR and between 4f91c20 and 483b411.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (6)
  • .gitignore
  • cmd/import/main.go
  • go.mod
  • internal/domain/entities/language.go
  • internal/mapper/scan_mapper_impl.go
  • internal/repository/scan_repository_qdrant_impl.go
✅ Files skipped from review due to trivial changes (1)
  • .gitignore
🚧 Files skipped from review as they are similar to previous changes (3)
  • internal/mapper/scan_mapper_impl.go
  • internal/repository/scan_repository_qdrant_impl.go
  • cmd/import/main.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/qdrant-restore-snapshots.sh`:
- Around line 56-57: The curl invocation in scripts/qdrant-restore-snapshots.sh
is manually setting the header "-H 'Content-Type:multipart/form-data'" while
also using curl's -F "snapshot=@${file}", which can omit the required multipart
boundary; remove the explicit Content-Type header and let curl set it
automatically so the -F upload uses the correct multipart boundary (locate the
curl command that includes "-H 'Content-Type:multipart/form-data'" and the -F
"snapshot=@${file}" options and delete the -H header).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bc918435-2d61-4827-92a5-90fe5a28a12a

📥 Commits

Reviewing files that changed from the base of the PR and between 483b411 and c9fac77.

📒 Files selected for processing (4)
  • README.md
  • scripts/README.md
  • scripts/qdrant-generate-snapshots.sh
  • scripts/qdrant-restore-snapshots.sh
✅ Files skipped from review due to trivial changes (1)
  • scripts/README.md

Comment thread scripts/qdrant-restore-snapshots.sh

@scanoss-qg scanoss-qg left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mscasso-scanoss
mscasso-scanoss merged commit 5c83125 into main Jun 5, 2026
4 checks passed
@mscasso-scanoss
mscasso-scanoss deleted the mscasso/feat/add-urlhash-and-lincese-to-report-SP4189 branch June 5, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants