Skip to content

Repair post-agent state push retries and expose git errors - #67474

Merged
pelikhan merged 6 commits into
mainfrom
copilot/aw-top-10-08-fix-job-failures
Oct 10, 2026
Merged

pelikhan merged 6 commits into
mainfrom
copilot/aw-top-10-08-fix-job-failures

Conversation

Copilot AI commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Post-agent ledger, repo-memory, and evaluation-state jobs repeatedly fail, while failure issues omit the underlying errors. The inspected reports have distinct causes, not one confirmed push-conflict bug.

  • Retry recovery: Restore command-scoped credentials for evaluation-state retry fetches; advance the retry base only after fetching succeeds.
  • Failure diagnostics: Preserve and redact git push stdout/stderr. Include bounded, sanitized error excerpts in state-push failure issues; retain job links when logs are unavailable.
  • Regression coverage: Exercise real non-fast-forward recovery while preserving concurrent JSONL rows, authenticated retries, and persistent git-error reporting.

Copilot AI linked an issue Oct 10, 2026 that may be closed by this pull request
2 tasks
Copilot AI and others added 2 commits October 10, 2026 16:47
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix post-agent state push job failures Repair post-agent state push retries and expose git errors Oct 10, 2026
Copilot AI requested a review from pelikhan October 10, 2026 16:50
@pelikhan
pelikhan marked this pull request as ready for review October 10, 2026 17:52
Copilot AI balanced review requested due to automatic review settings October 10, 2026 17:52
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #67474

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

No ADR enforcement needed for PR #67474: no "implementation" label and 0 new lines in business logic directories (threshold 100, 7 files changed, default config).

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ponytail review:

  • actions/setup/js/report_failed_jobs.cjs:L26: one redundant response-type gate can be removed.

net: -3 lines possible.

Generated by ✂️ Ponytail Reviewer for #67474 · codex · gpt56 · 10.9 AIC · ⌖ 5.07 AIC · ⊞ 13.5K
Comment /ponytail to run again

Comment thread actions/setup/js/report_failed_jobs.cjs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The retry, redaction, diagnostic, and regression-test changes are coherent and have no identified correctness issues.

0 open findings

What changed in this PR

Improves post-agent state push reliability and diagnostics.

Changes:

  • Authenticates evaluation-state retry fetches and updates retry bases safely.
  • Preserves, redacts, and reports git push errors.
  • Adds regression coverage for conflicts, credential handling, and diagnostics.
File Description
actions/​setup/​js/​report_failed_jobs.test.cjs Tests state-push error reporting and redaction.
actions/​setup/​js/​report_failed_jobs.cjs Adds sanitized git-error excerpts to failure issues.
actions/​setup/​js/​push_signed_commits.test.cjs Tests push-error capture and credential redaction.
actions/​setup/​js/​push_signed_commits.cjs Captures and sanitizes git push output.
actions/​setup/​js/​push_repo_memory.test.cjs Tests real non-fast-forward recovery.
actions/​setup/​js/​push_experiment_state.test.cjs Tests authenticated retry fetches.
actions/​setup/​js/​push_experiment_state.cjs Restores retry authentication and safe base advancement.

🧠 Review effort: Balanced


💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

@github-actions github-actions Bot mentioned this pull request Oct 10, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking concern

The new state-push diagnostics path decodes binary job-log payloads as UTF-8 text without unpacking them first, so real Actions log downloads can collapse into garbage and still omit the git failure details this PR is meant to surface.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 73.5 AIC · ⌖ 5.24 AIC · ⊞ 19.5K
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs, /tdd, and /codebase-design (triage: bug_fix). Both fixes address real root causes with targeted regression tests — no blocking issues, two non-blocking improvements flagged inline.

📋 Key Themes & Highlights

Key Themes

  • Redaction-logic duplication (/codebase-design): the URL/Authorization-header redaction regex pair is now copy-pasted in push_signed_commits.cjs, report_failed_jobs.cjs, and (in a slightly different form) push_repo_memory.cjs. Worth consolidating into one helper to avoid drift.
  • Untested error path (/tdd): the ls-remote failure branch in push_experiment_state.cjs's retry loop is silently swallowed with no test or log line, unlike the sibling fetch-failure branch which does log.

Positive Highlights

  • ✅ Root-cause fix is correct: currentBaseRef now only advances after a successful fetch, closing the stale-baseRef race that caused repeated non-fast-forward retries.
  • ✅ Retry fetches are now authenticated via getGitAuthEnv, fixing a real credential gap.
  • ✅ push_signed_commits.cjs switches from exec.exec to exec.getExecOutput so push failures expose real git stderr instead of just an exit code — paired with solid redaction tests (credential leakage, stderr-only exec wrappers).
  • ✅ report_failed_jobs.cjs adds bounded, sanitized git-error excerpts to state-push failure issues, with tests for missing logs, binary log buffers, and Markdown-fence neutralization.
  • ✅ The push_repo_memory.test.cjs rename from "compare-and-swap loss" to "non-fast-forward" test name more accurately reflects what's being exercised, and now exercises a real git push conflict rather than a mocked GraphQL error.

Investigation Note

Reviewed the full diff directly (379-line patch, well within budget); no areas of remaining uncertainty.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 117.5 AIC · ⌖ 14.7 AIC · ⊞ 10.3K
Comment /matt to run again

Comments that could not be inline-anchored

actions/setup/js/push_signed_commits.cjs:282

[/codebase-design] This redaction pipeline (built-in patterns + secret list + two ad-hoc URL/Authorization regex replaces) is now duplicated almost verbatim in report_failed_jobs.cjs (lines 31-33) and partially in push_repo_memory.cjs (redactFailureSummary). A third near-identical copy makes the interface shallow — any future fix to the credential-masking regex has to be applied in three places.

<details>
<summary>💡 Suggested consolidation</summary>

Extract a single `redactGitOutpu…

actions/setup/js/push_experiment_state.cjs:576

[/tdd] Good fix: currentBaseRef now only advances after fetch succeeds (lines 567-573), closing the stale-baseRef bug. But the outer catch {} (line 576) that swallows an ls-remote failure has no test — if ls-remote itself fails on a retry (not just fetch), the loop silently falls through to the same pushSignedCommits retry with a stale baseRef, with zero diagnostic trail.

<details>
<summary>💡 Suggested test / fix</summary>

Add a case to the it.each block (or a new test)…

Update push-handler mocks to return captured git exit codes and stderr, preserving non-fast-forward fallback and deleted-branch assertions. Surface ls-remote retry failures and verify plain-text job logs across supported byte formats.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (actions/setup/js/report_failed_jobs.cjs:26): L26: shrink: response-type guard that only throws into the immediately caught fallback. Buffer.from(response.data).toString("utf8") lets conversion failures take that same path. - Repair post-agent state push retries and expose git errors #67474 (comment)
  3. Fix failing check JS Tests (shard 3/4) (FAILURE): https://github.com/github/gh-aw/actions/runs/38069421591/job/114275813109.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: d704c32
Sous-chef work: 0491a534bdda6df57e666342c7988cd942957b36c663637456844123b0331e92 99fc195661b90dc631a5960c0642aa21aa4812000fe92377f25629fbdc6e618e
Sous-chef state: 7b18d6457c188ade709935473f9038caba655532a1c5a6a28eba3a7f6363bb4b

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 7.46 AIC · ⌖ 20.5 AIC · ⊞ 1K · ◷
Comment /souschef to run again

…fix-job-failures

Co-authored-by: gh-aw-bot <4175913+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Merged latest main in 53bc88f. The reported JS failures were fixed in 091a10b; all 351 affected tests pass locally. The response-type review thread already has a substantive reply and is resolved. Formatting, lint, build, and impacted tests pass. CI is stale after the merge and needs a maintainer re-trigger.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot ensure no git output is stored in log files, it can only go to the actions core.* which gets redacted. Otherwise there is a risk of leaking the git token.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

@copilot ensure no git output is stored in log files, it can only go to the actions core.* which gets redacted. Otherwise there is a risk of leaking the git token.

Removed repo-memory git diagnostics from the step-summary file and captured git subprocess output so reporting goes through Actions logging APIs instead of raw process streams. Added regression coverage; focused validation passes. Commit: 1b76cde.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[AW Top 10] 08 Fix post-agent state push job failures

4 participants