Repository navigation
feat: improve NVD API & datafeed downloading to acknowledge and process downloads in order of completion - #8855
Merged
jeremylong merged 1 commit intoOct 30, 2026
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Reversed executor pool sizes misapply the configured download limit and processing concurrency cap.
1 open issue
What changed in this PR
This PR refactors NVD datafeed updates to handle downloads in completion order, avoiding waits on earlier downloads while preserving modified-feed processing last.
Changes:
- Use completion services to coordinate downloads and processing.
- Separate downloading from processor submission.
- Sort feed scheduling and mark modified-feed processors for deferred execution.
| File | Description |
|---|---|
| core/src/main/java/org/owasp/dependencycheck/data/update/NvdApiDataSource.java | Coordinates completion-order handling and processes the modified feed last. |
| core/src/main/java/org/owasp/dependencycheck/data/update/nvd/api/NvdApiProcessor.java | Adds a flag identifying processors that should be delayed. |
| core/src/main/java/org/owasp/dependencycheck/data/update/nvd/api/DownloadTask.java | Returns an unsubmitted processor after downloading. |
Review effort: Balanced
chadlwilson
force-pushed
the
process-feed-downloads-on-arrival
branch
from
October 22, 2026 07:03
ea1f822 to
4c61e02
Compare
chadlwilson
force-pushed
the
process-feed-downloads-on-arrival
branch
2 times, most recently
from
October 22, 2026 07:16
b600d37 to
8bca5fd
Compare
chadlwilson
marked this pull request as draft
October 22, 2026 07:45
chadlwilson
changed the title
chadlwilson
force-pushed
the
process-feed-downloads-on-arrival
branch
from
October 22, 2026 09:24
04b8beb to
3122eb3
Compare
chadlwilson
marked this pull request as ready for review
October 22, 2026 09:29
Collaborator
|
@chadlwilson mind fixing the conflicts? |
Collaborator
Author
Ack, will do. |
chadlwilson
marked this pull request as draft
October 30, 2026 14:30
chadlwilson
force-pushed
the
process-feed-downloads-on-arrival
branch
from
October 30, 2026 18:00
3122eb3 to
9fee7ca
Compare
Collaborator
Author
|
Unfortunately, the AI slop PR at #8819 made a mess of the logic here, and did overly complex things with the I've reworked it here. |
…ownloads as they complete Currently all downloads are processed only in the submitted order. There's no reason for this because each are for a separate year. We can acknowledge them and process them as they complete using a CompletionService, rather than using static ordering of Futures. Signed-off-by: Chad Wilson <29788154+chadlwilson@users.noreply.github.com>
chadlwilson
force-pushed
the
process-feed-downloads-on-arrival
branch
from
October 30, 2026 18:05
9fee7ca to
2b2ae52
Compare
chadlwilson
marked this pull request as ready for review
October 30, 2026 18:05
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Executor shutdown can leave writes running as the database closes, and overlapping modified and yearly downloads can cause older CVE data to overwrite newer records.
2 open issues
1 resolved since last review
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Change
Currently all datafeed downloads are processed and awaited in the submitted order; and the submitted order is based on hash, not logical order.
There's no functional reason they need to be processed in submission order because each feed other than the
modifiedone is for a separate year and subset of CVEs, and the submission order is also currently not consistent. We can acknowledge them and dispatch them for processing one-by-one as they complete using a CompletionService, rather than using static ordering of Futures.The current behaviour causes really weird stuff like sitting there waiting for slow 2002 and 2003 downloads when the 2004 downloads have already failed, which is quite frustrating when you're trying to debug a problematic proxy or very slow NVD (as is the state right now).
This change
modified, processed at the end)Related issues
N/A
Have test cases been added to cover the new functionality?
yes