Ignore Dependabot security updates in the failure watcher - #272
Conversation
📝 WalkthroughWalkthroughThe workflow documents Dependabot run title patterns and updates failure detection to inspect only version update runs from the last 8 days, excluding designated e2e runs. ChangesDependabot version failure watcher
Estimated code review effort: 2 (Simple) | ~10 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR updates the Dependabot Failure Watcher workflow to stop treating Dependabot security update workflow runs as actionable failures, so the weekly scheduled watcher reflects only failures that maintainers can realistically fix (Dependabot version-update runs).
Changes:
- Filters out Dependabot security-update runs based on
displayTitlepatterns (notably the rootin /. forsignature) and documented directory denylists. - Uses
gh run list --createdto bound the query server-side rather than fetching and filtering full history locally. - Expands and clarifies in-file documentation explaining the different kinds of “Dependabot Updates” runs and why specific title substrings are load-bearing.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
GitHub runs both Dependabot version updates and Dependabot security
updates under one workflow name, "Dependabot Updates", and the watcher
counted both. Security updates routinely fail for reasons no pull request
can fix -- the advisory is against a dependency this project does not
declare directly, or no patched version is reachable. Left alone the
watcher stays red every week on those and trains everyone to ignore it.
Filter those runs out by title. Version updates are unaffected.
The title check is subtler than it looks, so document it properly. A
security job is marked by "/." AND a " for " suffix together; version
updates are either "/." with no " for " (the scheduled scan) or "/" with
one (the pull request). Both halves of " in /. for " are therefore
load-bearing -- matching on " in /." alone would discard every scan run,
which is most of the version-update runs and the shape the failures this
watcher was written for actually took.
That "/." spelling only separates the two at the repo root. In a
subdirectory a security update and a version update's pull request render
identically, so drop " in /e2e/{js,ts} for " by name as well. The Node
repos this workflow is shared with carry committed lockfiles under e2e/js
and e2e/ts, whose transitive dev dependencies attract advisories no pull
request can fix, and nothing in either is shipped code. That is 16
unactionable failures in each of GeoIP2-node and minfraud-api-node over
retained history; repos without those directories are unaffected.
Unlike the root filter, this one is not free. Both Node repos configure
npm with directories: ["/", "**/*"], and that glob does match e2e/js and
e2e/ts, so those directories do get version updates -- there is an open
version-update pull request under e2e/ts in both repos as this is
written. Dropping the pattern discards their pull-request refresh
failures along with the security jobs, and the ecosystem label is no help
because Dependabot writes "npm_and_yarn" for both. Taken anyway: the
scheduled scan is what this watcher primarily exists to catch and is
still reported for those directories, so what is given up is the narrower
"one open pull request has gone stale" signal for two directories of test
scaffolding. After filtering, 4 genuine failures remain reported in
GeoIP2-node and 3 in minfraud-api-node.
Reading the directories out of dependabot.yml would look more general and
was the earlier plan here, but it fails green. Entries may use globs, and
minfraud-api-dotnet's directories: ["**/*"] yields titles like "nuget in
/**/*" for the scan and "nuget in /MaxMind.MinFraud for
System.Net.Http.Json" for the pull request, neither of which any literal
comparison against the configured value matches -- so its two real nuget
failures would have been dropped without a word. A stale denylist
re-introduces noise, which is loud; a stale allowlist hides failures.
Name the three kinds of run in the comment while here, because the
scheduled scan and the per-pull-request refresh are easy to conflate: the
refresh runs are one per open pull request and are triggered by pushes to
the base branch or by rebases, not by the schedule, so they arrive in
bursts after merges. The scan is the kind this watcher primarily exists
to catch, which is what makes hiding a hypothetical refresh failure under
e2e an acceptable cost rather than a hole.
Bound the query server-side with --created instead of fetching all of
history and filtering by date locally, so --limit now caps an
already-narrowed window rather than standing in for one, and the run list
drops from several API pages to one. --limit rises 100 -> 500 as a
backstop: it still applies before the title filter, and reaching it would
silently drop the oldest in-window runs.
This workflow is shared verbatim across MaxMind repos. The change was
developed in maxmind/device-android and is applied here unmodified; see
that repo's commit for the measurements it was derived from.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fe7be0d to
80021db
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/dependabot-failure-watcher.yml:
- Around line 104-113: Update the run collection around the gh run list
invocation so security runs are excluded before applying any effective result
cap; paginate through time slices until the complete eight-day window is
covered, or fail closed when retrieval is incomplete. Preserve the existing jq
filters for identifying version-update failures and ensure older failed runs
cannot be hidden by the 500-run fetch limit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 6b5a0be7-d628-4606-a4d4-920b9243eb1d
📒 Files selected for processing (1)
.github/workflows/dependabot-failure-watcher.yml
| runs=$(gh run list \ | ||
| --repo "$REPO" \ | ||
| --workflow "Dependabot Updates" \ | ||
| --limit 100 \ | ||
| --json conclusion,createdAt,displayTitle,url \ | ||
| --jq "[.[] | select((.conclusion == \"failure\" or .conclusion == \"startup_failure\" or .conclusion == \"timed_out\") and .createdAt >= \"$since\")]") | ||
| --created ">=$since" \ | ||
| --limit 500 \ | ||
| --json conclusion,createdAt,displayTitle,url) | ||
| failures=$(echo "$runs" | jq ' | ||
| [.[] | ||
| | select((.displayTitle | contains(" in /. for ")) | not) | ||
| | select((.displayTitle | test(" in /e2e/(js|ts) for ")) | not) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repository files matching dependabot watcher:"
fd -i 'dependabot-failure-watcher\.yml|dependabot.*watch' .github || true
echo
echo "Relevant workflow excerpt:"
if [ -f .github/workflows/dependabot-failure-watcher.yml ]; then
wc -l .github/workflows/dependabot-failure-watcher.yml
sed -n '80,130p' .github/workflows/dependabot-failure-watcher.yml | cat -n
fi
echo
echo "Search for run list pagination and related watchers:"
rg -n "gh run list|--limit|created|displayTitle|security|Dependabot Updates|all-clear|failure" .github/workflows || trueRepository: maxmind/MaxMind-DB
Length of output: 5562
🌐 Web query:
GitHub CLI gh run list --created --limit maximum number shown documentation
💡 Result:
The GitHub CLI gh run list command does not have a strict, hard-coded "maximum number" limit enforced by the CLI itself other than the integer limit supported by the --limit (or -L) flag [1][2]. The --limit flag controls the maximum number of workflow runs to fetch [1][3]. The current default value is 20 [1][2][4]. While you can specify higher values when using the flag, the actual number of runs returned is ultimately subject to the underlying GitHub API's constraints for pagination and request limits [5]. Regarding your query about the --created flag, it allows you to filter runs by their creation date [1][6]. It supports standard GitHub search syntax, meaning you can pass specific dates or date ranges (e.g., YYYY-MM-DD or YYYY-MM-DD..YYYY-MM-DD) to refine the results fetched within your specified limit [5][7]. For example, to fetch up to 100 recent runs created within a specific date range, you would use: gh run list --limit 100 --created "2026-01-01..2026-07-30" [5][7]
Citations:
- 1: https://cli.github.com/manual/gh_run_list
- 2: https://manpages.debian.org/testing/gh/gh-run-list.1.en.html
- 3: https://github.com/github/docs/blob/main/content/actions/how-tos/monitor-workflows/view-workflow-run-history.md
- 4: Improve
run listdoc with available--jsonfields cli/cli#8934 - 5: Allow filtering workflow runs by date range in gh run list cli/cli#9042
- 6: gh run list use of status is confusing blend of status and conclusion cli/cli#8106
- 7: Improve help text for --created flag in gh run list cli/cli#13043
Do not apply the fetch limit before excluding security runs.
--limit 500 is applied before the local jq filters, so a burst of security Dependabot runs can fill the fetched result set and hide older failed version-update runs. If limit pagination returns only --limit runs, this can incorrectly report all-clear. Paginate the time slices until the full eight-day window is covered, or fail closed when the full in-window run set is not retrieved.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/dependabot-failure-watcher.yml around lines 104 - 113,
Update the run collection around the gh run list invocation so security runs are
excluded before applying any effective result cap; paginate through time slices
until the complete eight-day window is covered, or fail closed when retrieval is
incomplete. Preserve the existing jq filters for identifying version-update
failures and ensure older failed runs cannot be hidden by the 500-run fetch
limit.
Source: MCP tools
GitHub runs Dependabot version updates and Dependabot security updates under one
workflow name,
Dependabot Updates, and the watcher counted both. Securityupdates routinely fail for reasons no pull request can fix -- the advisory is
against a dependency the project does not declare directly, or no patched
version is reachable -- so the watcher stays red every week on those and trains
everyone to ignore it.
This filters security-update runs out by title, documents why both halves of
" in /. for "are load-bearing, and bounds thegh run listquery server-sidewith
--createdinstead of fetching all of history and filtering locally.See the commit message for the full reasoning, including why reading the
directory list out of
dependabot.ymlwas tried and rejected.This workflow is shared verbatim across MaxMind repos. The change was developed
in maxmind/device-android (maxmind/device-android#71)
and is applied here unmodified; the resulting file is byte-identical in every
repo.
Summary by CodeRabbit