Add support for introduced and fixed commits in AdvisoryData #2017

ziadhany · 2025-11-03T15:13:11Z

Introduce introduced_by_commits and fixed_by_commits fields in our advisory
Update from_dict and to_dict methods
Create a CodePatchData importer class
Enhance code commit collection capabilities for VCIO #2022

TG1999 · 2025-11-06T08:30:52Z

@ziadhany add description in the PR please!

vulnerabilities/importer.py

vulnerabilities/importers/osv.py

vulnerabilities/importer.py

TG1999 · 2025-11-07T07:32:48Z

@ziadhany mostly looks good! Please run the importer once and paste the logs here. Thanks!

I want to see if we are missing on any data in OSV format. And how does the AdvisoryData and ImpactedPackages looks with the new CommitData. Thanks!

vulnerabilities/importers/osv.py

ziadhany · 2025-11-07T14:56:48Z

@TG1999 This is the log output for the following importers:

pysec_importer_v2
pypa_importer_v2
oss_fuzz_importer_v2

importers_logs.zip

the database query result :
vulnerabilities_advisoryv2 Total rows: 10274
vulnerabilities_impactedpackage_fixed_by_commits Total rows: 4013
vulnerabilities_impactedpackage_affecting_commits Total rows: 3623
vulnerabilities_codecommit Total rows: 3791

TG1999 · 2025-11-10T17:11:03Z

@ziadhany

Invalid VersionRange  for affected_pkg: {'package': {'name': 'apache-commons-io', 'ecosystem': 'OSS-Fuzz', 'purl': 'pkg:generic/apache-commons-io'}, 'ranges': [{'type': 'GIT', 'repo': 'https://github.com/apache/commons-io.git', 'events': [{'introduced': '72b1f88fb722def136ce87c9b2bfdd3c9126bb3d'}, {'fixed': 'd3e5bd6de8bc96abbadccea8b934dc038a32e90c'}]}], 'versions': ['commons-io-2.14.0-RC1', 'rel/commons-io-2.14.0'], 'ecosystem_specific': {'severity': 'LOW'}, 'database_specific': {'introduced_range': 'c511d15294d1a406a177368804014313948e2601:06fde31494c279ad940149e1a3d4944040c73c0d', 'fixed_range': '247c8e7d85a8df293011c7e9c94fd50bb2986fb7:d3e5bd6de8bc96abbadccea8b934dc038a32e90c'}} for OSV id: 'OSV-2023-962': error:InvalidVersion("'commons-io-2.14.0-RC1' is not a valid <class 'univers.versions.SemverVersion'>")
Invalid VersionRange  for affected_pkg: {'package': {'name': 'apache-commons-io', 'ecosystem': 'OSS-Fuzz', 'purl': 'pkg:generic/apache-commons-io'}, 'ranges': [{'type': 'GIT', 'repo': 'https://github.com/apache/commons-io.git', 'events': [{'introduced': '72b1f88fb722def136ce87c9b2bfdd3c9126bb3d'}, {'fixed': 'd3e5bd6de8bc96abbadccea8b934dc038a32e90c'}]}], 'versions': ['commons-io-2.14.0-RC1', 'rel/commons-io-2.14.0'], 'ecosystem_specific': {'severity': 'LOW'}, 'database_specific': {'introduced_range': 'c511d15294d1a406a177368804014313948e2601:06fde31494c279ad940149e1a3d4944040c73c0d', 'fixed_range': '247c8e7d85a8df293011c7e9c94fd50bb2986fb7:d3e5bd6de8bc96abbadccea8b934dc038a32e90c'}} for OSV id: 'OSV-2023-618': error:InvalidVersion("'commons-io-2.14.0-RC1' is not a valid <class 'univers.versions.SemverVersion'>")

Why are we getting in this logs? The commit data should have been created for this

TG1999 · 2025-11-10T17:13:08Z

See all Invalid VersionRange errors. Why these are coming?

{'package': {'name': 'apache-commons-codec', 'ecosystem': 'OSS-Fuzz', 'purl': 'pkg:generic/apache-commons-codec'}, 'ranges': [{'type': 'GIT', 'repo': 'https://gitbox.apache.org/repos/asf/commons-codec.git', 'events': [{'introduced': '44e4c4d778c3ab87db09c00e9d1c3260fd42dad5'}, {'fixed': '3bf874e2141dc08550c0b330c7a7006f358bb0f0'}]}], 'versions': ['commons-codec-1.16.1-RC1', 'rel/commons-codec-1.16.1'], 'ecosystem_specific': {'severity': 'LOW'}, 'database_specific': {'fixed_range': '72c40fe6f62410bcaa019dbf2cb570ee4e49b70e:3bf874e2141dc08550c0b330c7a7006f358bb0f0'}} for OSV id: 'OSV-2023-1195': error:InvalidVersion("'commons-codec-1.16.1-RC1' is not a valid <class 'univers.versions.SemverVersion'>")

when we have introduced and fixed events to create code commit data.

ziadhany · 2025-11-11T02:43:52Z

I updated the script to handle unsupported packages (especially for OSS-Fuzz). CodeCommit is no longer ignored even if the package is unsupported, and logs are now more meaningful.

This is the updated logs:
importers_v2.zip

the database query result :
vulnerabilities_advisoryv2 Total rows: 17041
vulnerabilities_impactedpackage_fixed_by_commits Total rows: 7343
vulnerabilities_impactedpackage_affecting_commits Total rows: 6553
vulnerabilities_codecommit Total rows: 6553

Issues related:

pysec_importer_v2 / pypa_importer_v2:
- Invalid PypiVersion univers#174
- Collect OSV Malicious packages #2019
oss_fuzz_importer_v2
- Unsupported package type: None in OSV: 'OSV-2021-1227' This means the package type is unknown (e.g., generic, etc.), and there is no PURL associated with it.
- Invalid VersionRange for affected_pkg It depends on whether this is a valid version, for example, a semver version or not.
  example:
  > SemverVersion('commons-io-2.14.0-RC1')
  > univers.versions.InvalidVersion: 'commons-io-2.14.0-RC1' is not a valid <class 'univers.versions.SemverVersion'>

TG1999 · 2025-11-11T15:04:45Z

ERROR 2025-11-11 13:34:49.213781 UTC Unsupported PyPI advisory data file: GHSA-227r-w5j2-6243.json

This log does not tell me a lot, what's the data. Why this is unsupported.

TG1999 · 2025-11-11T15:05:57Z

Invalid VersionRange for affected_pkg: ['0.8', '0.9', '0.9.3', '0.9.4', '0.9.5', '0.9.6', '0.9.7', '0.9.8', '0.9.9', '2.0.1', '2.0.1rc1', '2.0.1rc2-git', '2.0.1rc3', '2.0.1rc4', '2.0.2', '2.0.3', '2.0.4', '2.0.5', '2.0b4', '2.0b5', '2.0b6', '2.0b7', '2.0b8', '2.0b9', '3.0.0', '3.0.0b1', '3.0.0b2', '3.0.1', '3.0.2', '3.0.3', '3.0.4', '3.0.5', '3.1', '3.2', '3.2.1', '3.2.2', '3.2.3', '3.2.4', '3.2.5', '3.3', '3.4', '3.4.1', '3.4.2', '3.4.3', '3.4.4', '3.4.5', '3.5', '3.5b1', '3.6', '3.6.1', '3.6.2', '3.6.3', '3.6.4'] for OSV id: 'PYSEC-2021-859': error:InvalidVersion("'2.0.1rc2-git' is not a valid <class 'univers.versions.PypiVersion'>")

One of the list might not be a valid version, but all others are valid, are we ingesting them or skipping whole list if we can't ingest one.

ziadhany · 2025-11-11T15:09:06Z

ERROR 2025-11-11 13:34:49.213781 UTC Unsupported PyPI advisory data file: GHSA-227r-w5j2-6243.json

This log does not tell me a lot, what's the data. Why this is unsupported.

@TG1999 We are ignoring GHSA files since we target only PYSEC files.
https://github.com/aboutcode-org/vulnerablecode/blob/main/vulnerabilities/pipelines/v2_importers/pysec_importer.py#L54

TG1999 · 2025-11-11T15:14:40Z

ERROR 2025-11-11 13:34:49.213781 UTC Unsupported PyPI advisory data file: GHSA-227r-w5j2-6243.json

This log does not tell me a lot, what's the data. Why this is unsupported.

@TG1999 We are ignoring GHSA files since we target only PYSEC files.

https://github.com/aboutcode-org/vulnerablecode/blob/main/vulnerabilities/pipelines/v2_importers/pysec_importer.py#L54

Then add that to the log as well :)

ziadhany · 2025-11-11T15:21:25Z

Invalid VersionRange for affected_pkg: ['0.8', '0.9', '0.9.3', '0.9.4', '0.9.5', '0.9.6', '0.9.7', '0.9.8', '0.9.9', '2.0.1', '2.0.1rc1', '2.0.1rc2-git', '2.0.1rc3', '2.0.1rc4', '2.0.2', '2.0.3', '2.0.4', '2.0.5', '2.0b4', '2.0b5', '2.0b6', '2.0b7', '2.0b8', '2.0b9', '3.0.0', '3.0.0b1', '3.0.0b2', '3.0.1', '3.0.2', '3.0.3', '3.0.4', '3.0.5', '3.1', '3.2', '3.2.1', '3.2.2', '3.2.3', '3.2.4', '3.2.5', '3.3', '3.4', '3.4.1', '3.4.2', '3.4.3', '3.4.4', '3.4.5', '3.5', '3.5b1', '3.6', '3.6.1', '3.6.2', '3.6.3', '3.6.4'] for OSV id: 'PYSEC-2021-859': error:InvalidVersion("'2.0.1rc2-git' is not a valid <class 'univers.versions.PypiVersion'>")

One of the list might not be a valid version, but all others are valid, are we ingesting them or skipping whole list if we can't ingest one.

We are skipping this since the version range would likely be inconsistent if we processed it.
I also created a related issue in univers:

Invalid PypiVersion univers#174

I can changes this if needed.

TG1999 · 2025-11-11T15:44:09Z

Invalid VersionRange for affected_pkg: ['0.8', '0.9', '0.9.3', '0.9.4', '0.9.5', '0.9.6', '0.9.7', '0.9.8', '0.9.9', '2.0.1', '2.0.1rc1', '2.0.1rc2-git', '2.0.1rc3', '2.0.1rc4', '2.0.2', '2.0.3', '2.0.4', '2.0.5', '2.0b4', '2.0b5', '2.0b6', '2.0b7', '2.0b8', '2.0b9', '3.0.0', '3.0.0b1', '3.0.0b2', '3.0.1', '3.0.2', '3.0.3', '3.0.4', '3.0.5', '3.1', '3.2', '3.2.1', '3.2.2', '3.2.3', '3.2.4', '3.2.5', '3.3', '3.4', '3.4.1', '3.4.2', '3.4.3', '3.4.4', '3.4.5', '3.5', '3.5b1', '3.6', '3.6.1', '3.6.2', '3.6.3', '3.6.4'] for OSV id: 'PYSEC-2021-859': error:InvalidVersion("'2.0.1rc2-git' is not a valid <class 'univers.versions.PypiVersion'>")

One of the list might not be a valid version, but all others are valid, are we ingesting them or skipping whole list if we can't ingest one.

We are skipping this since the version range would likely be inconsistent if we processed it.

I also created a related issue in univers:

Invalid PypiVersion univers#174

I can changes this if needed.

@keshav-space @pombredanne thoughts on this one ?

TG1999 · 2025-11-11T16:44:07Z

For PYSEC data we would be using github version range, coz the versions are Semver. And if a version is not parsable that version should be skipped. Not the entire range. Also we should introduce a flag for advisories that were not completely parsed. So in future if our parsing techniques gets better we can delete the incomplete parsed advisory with a new one.

keshav-space

Thanks @ziadhany, See the comments below and also make sure to adjust the insert_advisory method accordingly.

vulnerabilities/importer.py

ziadhany · 2025-11-17T14:33:36Z

This is the log output for the following importers:

pysec_importer_v2
pypa_importer_v2
oss_fuzz_importer_v2
github_osv_importer_v2

importers.zip

TG1999 · 2025-11-17T14:35:02Z

Failed to extract fixed commits: ValueError('Commit must be a valid a commit_hash.') We need to know the hash here. Do log the hash as well.

vulnerabilities/models.py

keshav-space

Thanks @ziadhany, LGTM!

TG1999 · 2025-12-15T17:29:42Z

@ziadhany LGTM! please rebase and adjust the migrations! great work 🙌

Signed-off-by: ziad hany <[email protected]>

Fix patch_checksum constraint Remove unused imports Signed-off-by: ziad hany <[email protected]>

Signed-off-by: ziad hany <[email protected]>

…PackageCommitPatch Signed-off-by: ziad hany <[email protected]>

Signed-off-by: ziad hany <[email protected]>

Update get_or_create_advisory_references to store the reference type correctly. Update get_or_create_advisory_package_commit_patches to correctly create or update the patch_text field. Signed-off-by: ziad hany <[email protected]>

Signed-off-by: ziad hany <[email protected]>

Add constraint to make sure we have at least one field to create a valid Patch obj. Update patch_text only if patch_text field is empty. Return multiple objects for classify_patch_source function Add patch in AdviosryData.from_dict() Signed-off-by: ziad hany <[email protected]>

Signed-off-by: ziad hany <[email protected]>

Update migration file Signed-off-by: ziad hany <[email protected]>

ziadhany · 2025-12-15T19:17:11Z

@ziadhany LGTM! please rebase and adjust the migrations! great work 🙌

@TG1999 Done , Please merge 🚀

TG1999

LGTM!

ziadhany force-pushed the advisory-fix-commit-1 branch from 2af10cf to a8ec9f1 Compare November 4, 2025 15:58

ziadhany marked this pull request as ready for review November 4, 2025 16:01

ziadhany changed the title ~~Add support for affected_by_commits and fixed_by_commits~~ Add support for affected_by_commits, fixed_by_commits, and OSV code fix commits Nov 5, 2025

ziadhany requested review from TG1999 and keshav-space and removed request for keshav-space November 5, 2025 15:40

TG1999 reviewed Nov 6, 2025

View reviewed changes

vulnerabilities/importer.py Outdated Show resolved Hide resolved

TG1999 reviewed Nov 6, 2025

View reviewed changes

vulnerabilities/importers/osv.py Outdated Show resolved Hide resolved

TG1999 reviewed Nov 6, 2025

View reviewed changes

vulnerabilities/importer.py Show resolved Hide resolved

ziadhany requested a review from TG1999 November 7, 2025 02:48

TG1999 reviewed Nov 7, 2025

View reviewed changes

vulnerabilities/importers/osv.py Outdated Show resolved Hide resolved

ziadhany requested a review from TG1999 November 7, 2025 14:56

keshav-space requested changes Nov 13, 2025

View reviewed changes

vulnerabilities/importer.py Outdated Show resolved Hide resolved

vulnerabilities/importer.py Outdated Show resolved Hide resolved

TG1999 mentioned this pull request Nov 14, 2025

Enhance code commit collection capabilities for VCIO #2022

Open

keshav-space requested changes Dec 11, 2025

View reviewed changes

vulnerabilities/models.py Outdated Show resolved Hide resolved

keshav-space requested changes Dec 11, 2025

View reviewed changes

vulnerabilities/models.py Outdated Show resolved Hide resolved

ziadhany requested a review from keshav-space December 15, 2025 07:11

keshav-space approved these changes Dec 15, 2025

View reviewed changes

ziadhany closed this Dec 15, 2025

ziadhany deleted the advisory-fix-commit-1 branch December 15, 2025 17:55

ziadhany restored the advisory-fix-commit-1 branch December 15, 2025 17:56

ziadhany reopened this Dec 15, 2025

ziadhany added 15 commits December 15, 2025 20:56

Enhance code commit collection capabilities aboutcode-org#2022

e0a91d4

Signed-off-by: ziad hany <[email protected]>

Update aboutcode/federated to work with packageurl-python 0.17.6

22ee395

Signed-off-by: ziad hany <[email protected]>

Update aosp importer

6625782

Fix patch_checksum constraint Remove unused imports Signed-off-by: ziad hany <[email protected]>

Fix compute_patch_checksum typo

435a4ad

Signed-off-by: ziad hany <[email protected]>

Update the model to use compute_patch_checksum function

a42b935

Signed-off-by: ziad hany <[email protected]>

Update classify_patch_source function so we can store both Patch and …

061dbbf

…PackageCommitPatch Signed-off-by: ziad hany <[email protected]>

Update the model to store commit hash if unsupported

e591edb

Signed-off-by: ziad hany <[email protected]>

Add an end-to-end test for patch advisory creation.

b8b99d3

Update get_or_create_advisory_references to store the reference type correctly. Update get_or_create_advisory_package_commit_patches to correctly create or update the patch_text field. Signed-off-by: ziad hany <[email protected]>

Update migration file

9e8a589

Signed-off-by: ziad hany <[email protected]>

Add a test for intorduced/fixed PackageCommitPatch

f1bcb81

Signed-off-by: ziad hany <[email protected]>

Update Patch model constraint to be database constraints.

eb2d063

Signed-off-by: ziad hany <[email protected]>

Update Patch constraint to handel empty and null case

64f43d7

Signed-off-by: ziad hany <[email protected]>

Update Patch constraint and add a test for the constraint

36c4174

Signed-off-by: ziad hany <[email protected]>

Resolve merge conflict

f7ee8c2

Update migration file Signed-off-by: ziad hany <[email protected]>

ziadhany force-pushed the advisory-fix-commit-1 branch from 747dc5f to f7ee8c2 Compare December 15, 2025 19:13

TG1999 approved these changes Dec 15, 2025

View reviewed changes

TG1999 merged commit 32d9724 into aboutcode-org:main Dec 15, 2025
5 of 6 checks passed

ziadhany mentioned this pull request Dec 15, 2025

Collect existing fix commits for aosp_dataset #1988

Closed

ziadhany deleted the advisory-fix-commit-1 branch December 15, 2025 19:38

Uh oh!

Add support for introduced and fixed commits in AdvisoryData #2017

Add support for introduced and fixed commits in AdvisoryData #2017

Uh oh!

Conversation

ziadhany commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TG1999 commented Nov 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TG1999 commented Nov 7, 2025

Uh oh!

Uh oh!

ziadhany commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TG1999 commented Nov 10, 2025

Uh oh!

TG1999 commented Nov 10, 2025

Uh oh!

ziadhany commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TG1999 commented Nov 11, 2025

Uh oh!

TG1999 commented Nov 11, 2025

Uh oh!

ziadhany commented Nov 11, 2025

Uh oh!

TG1999 commented Nov 11, 2025

Uh oh!

ziadhany commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TG1999 commented Nov 11, 2025

Uh oh!

TG1999 commented Nov 11, 2025

Uh oh!

keshav-space left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ziadhany commented Nov 17, 2025

Uh oh!

TG1999 commented Nov 17, 2025

Uh oh!

Uh oh!

Uh oh!

keshav-space left a comment

Choose a reason for hiding this comment

Uh oh!

TG1999 commented Dec 15, 2025

Uh oh!

ziadhany commented Dec 15, 2025

Uh oh!

TG1999 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ziadhany commented Nov 3, 2025 •

edited

Loading

ziadhany commented Nov 7, 2025 •

edited

Loading

ziadhany commented Nov 11, 2025 •

edited

Loading

ziadhany commented Nov 11, 2025 •

edited

Loading