Scheduled drift detection: weekly gates + monthly notebooks re-run against fresh resolves - #1143
Scheduled drift detection: weekly gates + monthly notebooks re-run against fresh resolves#1143AlexanderFengler wants to merge 1 commit into
Conversation
- new drift.yml: weekly lint/fast/slow + monthly notebooks re-run the existing gates via workflow_call against a fresh dependency resolve of unchanged main (no lockfile is committed, so this is the upstream/ toolchain drift detector); failures file ONE deduped drift-labeled issue, green runs close it; force_fail dispatch input rehearses the issue path - run_slow_tests.yml + check_notebooks.yml gain workflow_call triggers - check_notebooks.yml: SKIP_NOTEBOOKS entries fixed to full find paths (the two workshop notebooks executed for months while listed as skipped: bare filenames never matched) and the run is split into a 4-way shard matrix with fail-fast off, so no shard can hit the 6-hour job limit and a drift run reports the full blast radius - drift fast job clears the PR-oriented pytest addopts (--exitfirst, --reruns) via test_args for honest triage output - setup-env composite actions: remove the dead cache-hit guards (no step with id 'cache' exists; the condition always evaluated true)
📝 WalkthroughWalkthroughThe CI setup actions now install dependencies unconditionally. Notebook checks support four-way sharding. Slow tests and notebook checks support reusable workflow calls. A scheduled Drift workflow runs checks and reports failures through GitHub issues. ChangesCI workflow orchestration
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant DriftWorkflow
participant ReusableChecks
participant ResultReporter
participant GitHubIssues
DriftWorkflow->>ReusableChecks: run configured CI checks
ReusableChecks-->>ResultReporter: return job results
ResultReporter->>GitHubIssues: close, comment on, or create drift issue
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/setup-env-notebooks/action.yml:
- Around line 29-30: Update the stale-guard explanation comments at
.github/setup-env-notebooks/action.yml lines 29-30 and
.github/setup-env/action.yml lines 29-30 to state that the nonexistent cache
step/output reference evaluated false and skipped dependency installation,
rather than claiming it evaluated true. No workflow logic changes are needed.
In @.github/workflows/drift.yml:
- Around line 53-56: Add a job-level concurrency configuration to the report job
that uses a stable repository-specific group and sets cancel-in-progress to
false, ensuring concurrent drift issue lookup-and-create runs queue and execute
serially.
- Around line 75-93: Update the gate-result evaluation loop in the drift
workflow so the issue-closing branch runs only when every gate result is
success, not merely when FAILED_JOBS is empty. Treat cancelled results as
non-success and preserve any existing drift issue; retain the current failure
reporting and force_fail behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5139c8e9-df07-4f74-aa50-fbb9b7a827dc
📒 Files selected for processing (5)
.github/setup-env-notebooks/action.yml.github/setup-env/action.yml.github/workflows/check_notebooks.yml.github/workflows/drift.yml.github/workflows/run_slow_tests.yml
| # (a stale `if: steps.cache.outputs.cache-hit` guard used to sit here — | ||
| # no step with id `cache` exists, so it always evaluated true) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
GitHub Actions documentation: How does an if expression evaluate when it references an output from a nonexistent step?
💡 Result:
In GitHub Actions, when an if expression references a nonexistent step or a nonexistent output from a step, it evaluates to an empty string [1]. Because GitHub Actions evaluates this reference as an empty string, the final outcome of the if condition depends on how that empty string is used in your expression [2]: - If you evaluate the reference directly as a boolean (e.g., if: steps.nonexistent.outputs.result), it effectively behaves as a falsy value, causing the step or job to be skipped [3][2]. - If you compare the reference against another value (e.g., if: steps.nonexistent.outputs.result == 'true'), the expression becomes an empty string compared against 'true', which evaluates to false [2]. This behavior is a common source of silent failures where steps or jobs are skipped without throwing a formal error [2]. It is important to note that the steps context only contains information about steps in the current job that have an ID specified and have already executed [1][4][5]. Referencing a step that has not yet run or does not exist results in this empty string evaluation [5].
Citations:
- 1: https://docs.github.com/en/actions/reference/workflows-and-actions/contexts
- 2: https://fixdevs.com/blog/github-actions-if-condition-not-working/
- 3: https://stackoverflow.com/questions/77141188/what-is-considered-truthy-or-falsy-in-expressions-contexts-in-github-actions
- 4: https://github.com/github/docs/blob/main/content/actions/reference/workflows-and-actions/contexts.md
- 5: https://nick2bad4u.github.io/eslint-plugin-github-actions-2/docs/rules/no-unknown-step-reference/
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Relevant action files:\n'
for f in .github/setup-env-notebooks/action.yml .github/setup-env/action.yml; do
if [ -f "$f" ]; then
echo "--- $f"
nl -ba "$f" | sed -n '1,70p'
fi
doneRepository: lnccbrown/HSSM
Length of output: 258
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for f in .github/setup-env-notebooks/action.yml .github/setup-env/action.yml; do
printf '\n--- %s\n' "$f"
if [ -f "$f" ]; then
awk '{printf "%6d %s\n", NR, $0}' "$f" | sed -n '1,70p'
else
printf 'missing\n'
fi
doneRepository: lnccbrown/HSSM
Length of output: 2531
Correct the stale-guard explanation.
In GitHub Actions, an if reference to a nonexistent step or missing output evaluates to an empty string, which is false. The old guard would have skipped dependency installation, so update both comments at lines 29-30 in .github/setup-env-notebooks/action.yml and .github/setup-env/action.yml to say the guard evaluated false/skipped the step rather than evaluated true.
📍 Affects 2 files
.github/setup-env-notebooks/action.yml#L29-L30(this comment).github/setup-env/action.yml#L29-L30
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/setup-env-notebooks/action.yml around lines 29 - 30, Update the
stale-guard explanation comments at .github/setup-env-notebooks/action.yml lines
29-30 and .github/setup-env/action.yml lines 29-30 to state that the nonexistent
cache step/output reference evaluated false and skipped dependency installation,
rather than claiming it evaluated true. No workflow logic changes are needed.
| report: | ||
| needs: [lint, fast, slow, notebooks] | ||
| if: always() | ||
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
GitHub Actions documentation: Does job-level concurrencyserialize jobs in the same group whencancel-in-progress is false?
💡 Result:
Yes, job-level concurrency serializes jobs within the same concurrency group, but it does not prevent the cancellation of queued jobs when cancel-in-progress is set to false [1][2]. GitHub Actions concurrency groups enforce a strict limit: at any given time, there can be at most one running job and one pending (queued) job per concurrency group [3][4][5]. When you use cancel-in-progress: false: 1. Currently running jobs are protected from being canceled [1]. 2. If a new job is triggered while another is already running and one is already pending, the new job will cancel the existing pending job to take its place in the queue [3][1][2]. In effect, cancel-in-progress: false ensures that the most recent job arrival becomes the "next in line" once the currently running job finishes, but it does not allow an unlimited queue of jobs to wait for execution [1][2]. If you require queuing more than one pending job, you must use the queue: max configuration (which supports up to 100 queued jobs) instead of relying solely on cancel-in-progress [5][2].
Citations:
- 1: https://dev.to/kanta13jp1/github-actions-concurrency-trap-cancel-in-progress-false-still-drops-queued-runs-5hg3
- 2: https://stackoverflow.com/questions/76096372/how-to-queue-more-than-one-github-action-workflow-run
- 3: https://docs.github.com/enterprise-cloud@latest/actions/using-jobs/using-concurrency
- 4: Clarify concurrency cancel-in-progress behaviour github/docs#30647
- 5: https://docs.github.com/actions/using-workflows/workflow-syntax-for-github-actions
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== file list =="
git ls-files '.github/workflows/*' | sort
echo
echo "== drift.yml outline =="
if [ -f .github/workflows/drift.yml ]; then
wc -l .github/workflows/drift.yml
ast-grep outline .github/workflows/drift.yml || true
echo
echo "== relevant drift.yml lines =="
nl -ba .github/workflows/drift.yml | sed -n '1,180p'
fi
echo
echo "== search for drift issue logic in workflow =="
rg -n "find|create|update|issue|drift|report" .github/workflows/drift.yml || trueRepository: lnccbrown/HSSM
Length of output: 735
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== relevant drift.yml lines =="
awk '{printf "%5d\t%s\n", NR, $0}' .github/workflows/drift.yml | sed -n '1,140p'
echo
echo "== concurrency declarations in workflows =="
python3 - <<'PY'
from pathlib import Path
for p in Path(".github/workflows").glob("*.yml"):
text = p.read_text()
if "concurrency:" in text:
print(f"\n--- {p} ---")
for line in text.splitlines():
if "concurrency:" in line.lower() or "cancel-in-progress:" in line.lower():
print(line)
PYRepository: lnccbrown/HSSM
Length of output: 5444
Serialize the drift-issue lookup-and-create step.
The report issue-creation step reads issues only at the start. If two report jobs are scheduled against the same repository while no drift issue exists, both can pass that check and create duplicate drift issues. Add a job-level concurrency group to report and use cancel-in-progress: false to serialize these runs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/drift.yml around lines 53 - 56, Add a job-level
concurrency configuration to the report job that uses a stable
repository-specific group and sets cancel-in-progress to false, ensuring
concurrent drift issue lookup-and-create runs queue and execute serially.
| # skipped is neutral: monthly-gated jobs are skipped in weekly runs | ||
| if [ "${!var}" = "failure" ]; then | ||
| FAILED_JOBS="$FAILED_JOBS $job" | ||
| fi | ||
| done | ||
| if [ "$FORCE_FAIL" = "true" ]; then | ||
| FAILED_JOBS="$FAILED_JOBS (force_fail rehearsal)" | ||
| fi | ||
|
|
||
| EXISTING=$(gh issue list --repo "$GITHUB_REPOSITORY" --label drift \ | ||
| --state open --search "in:body \"$KEY\"" \ | ||
| --json number --jq '.[0].number // empty') | ||
|
|
||
| if [ -z "$FAILED_JOBS" ]; then | ||
| if [ -n "$EXISTING" ]; then | ||
| gh issue close "$EXISTING" --repo "$GITHUB_REPOSITORY" \ | ||
| --comment "Scheduled drift checks green again: $RUN_URL" | ||
| fi | ||
| exit 0 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
GitHub Actions documentation: What values can needs.<job_id>.resultreturn, and how doesif: always() behave when a needed job is cancelled?
💡 Result:
In GitHub Actions, the needs.<job_id>.result property can return one of four possible string values: success, failure, cancelled, or skipped [1]. Regarding the behavior of if: always() when a needed job is cancelled: 1. Job Execution: When a job is configured with if: always(), it will evaluate to true even if the workflow is cancelled [2][3][4]. Consequently, the job will continue to run even if its dependencies (the needed jobs) are cancelled [2][5][3]. 2. Potential Issues: Because always() remains true upon cancellation, using it to guard steps or jobs can mask the true state of a workflow. If a job that should have been terminated continues to run due to always(), it may lead to unintended behavior, such as reporting a successful final job status despite earlier cancellations or failures [4]. 3. Recommended Alternatives: Official documentation and community best practices generally recommend using if: ${{ !cancelled() }} if you want a job or step to run regardless of success or failure, but still want it to be skipped if the workflow is cancelled [5]. For cleanup tasks that should only run on completion (and not necessarily cancellation), using if: success() || failure() is often preferred to ensure the job remains responsive to cancellation signals [4].
Citations:
- 1: https://docs.github.com/en/actions/reference/workflows-and-actions/contexts
- 2: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-cancellation
- 3: https://github.com/github/docs/blob/main/content/actions/reference/workflows-and-actions/workflow-cancellation.md
- 4: https://latchkey.dev/learn/github-actions/gha-if-always-masks-failures-in-ci
- 5: Provide a way to force-cancel actions and improve cancellation feedback actions/runner#1846
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files matching drift.yml:"
fd -a 'drift\.yml$' .github/workflows || true
echo
echo "Context around .github/workflows/drift.yml lines 1-130:"
if [ -f .github/workflows/drift.yml ]; then
nl -ba .github/workflows/drift.yml | sed -n '1,130p'
fi
echo
echo "Search for FAILED_JOBS and drift issue handling:"
rg -n "FAILED_JOBS|EXISTING|gh issue (close|list|create|reopen)|force_fail|Monthly|monthly-gated|cancelled|always\(\)" .github/workflows -S || true
echo
echo "Changed files/stat:"
git diff --name-status HEAD~1..HEAD 2>/dev/null | sed -n '1,120p' || trueRepository: lnccbrown/HSSM
Length of output: 323
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Context around .github/workflows/drift.yml lines 1-140:"
awk '{printf "%7d %s\n", NR, $0}' .github/workflows/drift.yml | sed -n '1,140p'
echo
echo "Search for FAILED_JOBS and drift issue handling:"
rg -n "FAILED_JOBS|EXISTING|gh issue (close|list|create|reopen)|force_fail|Monthly|monthly-gated|cancelled|always\(\)" .github/workflows -S || trueRepository: lnccbrown/HSSM
Length of output: 7339
🌐 Web query:
GitHub Actions workflow cancellation job cancelled result depends on workflow failure dependency cancelled skipped
💡 Result:
In GitHub Actions, workflow cancellation, job failures, and dependencies interact through specific status and conclusion states [1][2][3]. Workflow Cancellation When a workflow run is cancelled (e.g., manually or via concurrency settings), the GitHub server sends a cancellation signal to all running jobs [4]. - Running steps: The runner sends a SIGINT/Ctrl-C signal to processes. If they do not terminate within a timeout period (typically 7500ms + 2500ms), they are forcibly killed [4]. - Job status: Jobs that are effectively cancelled transition to a cancelled conclusion [1][2]. - Conditional execution: The cancelled() status check function evaluates to true only if the workflow run has been cancelled [4][2]. By default, jobs have an implicit if: success() condition, meaning they are skipped if a preceding dependency fails or is cancelled [4][2]. Dependencies and Statuses - Cancelled: Occurs when a workflow run is explicitly aborted [2]. - Failure: Occurs when at least one step in a job fails (non-zero exit code), provided the step did not have continue-on-error: true [2]. - Skipped: Occurs when a job's if condition evaluates to false [2]. Because jobs default to if: success(), if a dependency is cancelled or fails, downstream jobs are automatically skipped [2]. Key Considerations for Workflow Logic - Always Condition: Using if: always() forces a job to run regardless of the status of previous jobs or workflow cancellation, as it returns true even on cancellation [5][6]. - Handling Cancellation: To allow a job to run only when the workflow is cancelled, use if: cancelled(). To allow a job to run even if the workflow is cancelled (but not if it was skipped due to other conditions), consider using if: ${{ !cancelled() }} [5][6]. - Matrix/Reusable Workflow Nuance: In some complex scenarios, such as matrices calling reusable workflows, a failure in a child workflow might propagate a cancelled status to the parent, potentially causing the parent to report cancelled or even unexpected success, depending on configuration [7]. If a job is skipped unexpectedly, you can debug this by downloading the log archive from the workflow run and inspecting the system.txt file within the job logs, which details the evaluation of if conditions [5][6].
Citations:
- 1: https://docs.github.com/en/enterprise-cloud@latest/rest/actions/workflow-runs?apiVersion=2026-03-10
- 2: https://gist.github.com/kf6kjg/31b80cf0f8c9997ddc7fe7335c580412
- 3: https://github.com/actions/runner/blob/main/docs/adrs/0274-step-outcome-and-conclusion.md
- 4: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-cancellation
- 5: https://docs.github.com/en/actions/how-tos/troubleshoot-workflows
- 6: https://docs.github.com/en/enterprise-cloud@latest/actions/monitoring-and-troubleshooting-workflows/troubleshooting-workflows/about-troubleshooting-workflows
- 7:
cancelled()returnsfalseafter workflow cancel actions/runner#3041
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
job_results = {
"green": {"lint": "success", "fast": "success", "slow": "success", "notebooks": "skipped"},
"cancelled_downstream_run": {"lint": "success", "fast": "success", "slow": "success", "notebooks": "cancelled"},
"failure": {"lint": "failure", "fast": "success", "slow": "success", "notebooks": "skipped"},
}
for label, results in job_results.items():
failed_jobs = []
for job, result in results.items():
if result == "failure":
failed_jobs.append(job)
print(label, "=>", " ".join(failed_jobs) if failed_jobs else "no failed job")
PYRepository: lnccbrown/HSSM
Length of output: 233
Do not close the drift issue after a cancelled gate.
cancelled is a valid needs.<job>.result value, and this script only treats failure as red. If notebooks is cancelled, FAILED_JOBS stays empty and the if [ -z "$FAILED_JOBS" ] branch can close an open drift issue. Close the issue only when every gate is success; preserve the issue when a gate is cancelled.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/drift.yml around lines 75 - 93, Update the gate-result
evaluation loop in the drift workflow so the issue-closing branch runs only when
every gate result is success, not merely when FAILED_JOBS is empty. Treat
cancelled results as non-success and preserve any existing drift issue; retain
the current failure reporting and force_fail behavior.
1 participant
Part of the ecosystem self-healing rollout (spine PR lnccbrown/HSSMSpine#35 is the aggregation layer).
drift.yml: weekly (lint + fast + slow) and monthly (notebooks) scheduled re-runs of the existing gates viaworkflow_call, against a fresh PyPI resolve of unchanged main — with no committed lockfile this is the detector for upstream/toolchain releases breaking us (ruff 0.16 and pyrefly both did, silently, last month)drift-labeled issue; green runs close it;force_faildispatch input rehearses the pathrun_slow_tests.yml/check_notebooks.ymlgainworkflow_call:--exitfirst/--rerunsaddopts; deadcache-hitguards removed from the setup-env composite actions🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Improvements