fix(ci): re-run only failed specs on Windows test retries (PER-9011)

[AkashBrowserStack]

Problem

The Windows Test @percy/core check takes ~241 min, vs ~19 min for the same check on Linux (Linux even runs with coverage). This slows every PR's required Windows checks. Tracked in PER-9011.

Root cause

The 241 min is not one slow run — it's the same @percy/core suite run 4 times by the retry loop in .github/workflows/windows.yml:

Step	Duration	True outcome
Run tests	~62 min	failed → retried
Retry (1/4)	~59 min	failed → retried
Retry (2/4)	~59 min	failed → retried
Retry (3/4)	~59 min	passed (retry 4 skipped)

continue-on-error: true rewrites each failed step's conclusion to success, so the check looks green even though 3 of 4 attempts failed. Only ~1 spec out of 1078 flakes per run (browser/local-server timing on Windows) — and it's a different spec each time (e.g. Discovery > captures favicon when the server provides one, ... promise only for sync snapshot). Re-running the entire ~60-min suite to recover one flaky spec is the real waste.

Fix: make retries cheap instead of fighting every flaky spec

• scripts/test.js — the Jasmine node runner records failed spec names to PERCY_NODE_FAILURES_FILE; when run with PERCY_ONLY_FAILED_SPECS=1 it uses a specFilter to re-run only those specs. A filtered run reports incomplete (the rest are intentionally skipped), so success there means the targeted specs passed; full runs keep Jasmine's strict status. No behavior change when the env vars are unset (Linux/local/coverage all unaffected).
• .github/workflows/windows.yml — retry0 runs the full suite once; retries 1-4 set PERCY_ONLY_FAILED_SPECS=1 so they re-run only the spec(s) that just failed (seconds, not ~60 min).
• Honest check — a Flag flaky tests step emits a ::warning:: whenever a retry recovered a flake, so "green via retry" stays visible.

Verification (local, against the real @percy/core suite)

• Retry re-runs only the failed subset — proven on real failures: 28 specs in 12s vs the full 1078 specs in 1304s (~108× cheaper retry).
• Retry of passing spec → exit 0; retry of failing spec → exit 1 with the name recorded for the next retry.
• Caught and fixed a real bug along the way: filtered runs report incomplete, so the exit logic now treats "targeted specs passed" as success (otherwise every retry would have looked like a failure).

Expected impact

~241 min → ~62 min for @percy/core on Windows (one full pass + cheap retries), full coverage preserved.

The real Windows wall-clock is validated by this PR's Windows CI run. The ~3× single-pass gap (Windows ~60 vs Linux ~19 min) is intentionally out of scope here — it's a separate per-run-speedup follow-up.

Draft until Windows CI confirms the timing.

---

🤖 Generated with Claude Code

[rishigupta1599]

why only implement for windows?, lets do for linux as well will help in improvement overall

[AkashBrowserStack]

Good call — done in 9c81399. The runner support in scripts/test.js was already platform-agnostic; only the wiring was Windows-only, so I've extended it to test.yml (Linux) too. Linux flakes on the same timing-sensitive specs and had no retry at all.

One Linux-specific nuance worth flagging: Linux runs test:coverage with a 100% coverage gate (.nycrc), and a retry that re-runs only a few specs can't reach 100%. So I structured it as:

• retry0 → test:coverage (full suite + coverage gate, records any failed specs)
• retries 1–4 → plain test on only the failed specs, without the coverage gate (a subset can't hit 100%)
• if retry0 fails with no recorded spec failures (i.e. a genuine coverage drop or a crash, not a flaky spec), the runner now preserves that failure instead of masking it — so real coverage regressions still fail the build.

Verified locally before pushing: empty-failures retry → exit 1 (preserved); real-failure retry → runs only that spec → exit 0; and the coverage path correctly writes the failures file the retries consume. CI is re-running now.