Skip to content

TEST Stabilize merge-queue async and browser checks - #3011

Open
Roman Lutz (romanlutz) wants to merge 3 commits into
microsoft:mainfrom
romanlutz:romanlutz-merge-queue-failures
Open

Roman Lutz (romanlutz) wants to merge 3 commits into
microsoft:mainfrom
romanlutz:romanlutz-merge-queue-failures

Conversation

@romanlutz

@romanlutz Roman Lutz (romanlutz) commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

Merge-queue runs were rejecting unrelated PRs because short async-test watchdogs expired during setup or cleanup. Some watchdogs also sent an extra cancellation to the task under test, changing the cancellation count or original exception the test was meant to verify.

  • Expire real generator asyncio.Timeout contexts at the intended pending await, while checking the configured deadlines, cleanup, and original outcome.
  • Add a tested completion-wait helper that bounds observation without cancelling the operation under test. Give coordination waits more CI scheduling slack and release/drain owned estimate tasks in finally.
  • Apply the same watchdog handling to the SQLite cancellation tests newly landed in FIX SQLite cancellation cleanup #2982. The reported macOS failure occurred at the five-second inserted.wait() before cancellation was requested. Keep the rollback, connection cleanup, repeated-cancellation, and original-exception assertions intact for both database modes.
  • Remove the 200 ms press-and-hold from native-spinner single-step checks. Native Chromium spinners can auto-repeat while held; custom stepper tests still cover long clicks.
  • Document the async timing and cancellation test patterns.

No production code, production deadlines, coverage thresholds, retries, or required-check settings change. Cancellation, concurrency bounds, and health responsiveness while a database operation remains pending are still asserted. The separate Alembic repair for #2792 is not included.

Merged main at 13cb39257d normally. The generator-test conflict was resolved by preserving upstream setup/send assertions alongside this PR's controlled deadlines.

Tests and Documentation

Validation on Windows with all Python extras installed:

  • uv run --frozen --no-sync -m pytest -n 4 --dist=loadfile tests\unit --timeout=120 -q: passed, 23,725 tests, 21 skipped, and 84 subtests. Used the documented Makefile-equivalent command because make is unavailable. This full run preceded the later merges of main and SQLite watchdog follow-up.
  • uv run --frozen --no-sync pytest -q -n 4 --dist=loadfile tests\unit\test_async_utils.py tests\unit\memory\test_sqlite_cancellation.py tests\unit\memory\test_async_memory.py tests\unit\backend\test_main.py tests\unit\backend\test_scenario_service.py tests\unit\executor\promptgen\test_target_objective_generator.py tests\unit\scenario\core\test_scenario_partial_results.py --timeout=90: passed, 304 tests on the integrated branch including the SQLite follow-up.
  • uv run --frozen --no-sync pytest -q tests\unit\memory\test_sqlite_cancellation.py -k failed_session_rollback --timeout=90: passed, all 10 reported-path variants; 27 deselected.
  • Earlier integration check after merging main at f5c442d456: uv run --frozen --no-sync pytest -q -n 4 --dist=loadfile tests\unit\test_async_utils.py tests\unit\backend\test_main.py tests\unit\backend\test_scenario_service.py tests\unit\executor\promptgen\test_target_objective_generator.py tests\unit\scenario\core\test_scenario_partial_results.py tests\unit\score\test_scorer_metrics.py tests\unit\score\test_scorer_metrics_io.py --timeout=90: passed, 231 tests and 4 skipped.
  • From frontend, with an isolated E2E_FRONTEND_PORT: uv run --frozen --no-sync cmd.exe /c 'npm exec --no -- playwright test numeric-controls.spec.ts --project=mock --retries=0 --repeat-each=3 --workers=1 --fail-on-flaky-tests': passed, all 24 executions without retries, before the later main merge.
  • From frontend: npm exec --no -- eslint e2e\numeric-controls.spec.ts --max-warnings 0 and npm run type-check -- --pretty false: passed before the later main merge.
  • uv run --frozen --no-sync ruff check tests\unit\memory\test_sqlite_cancellation.py tests\unit\executor\promptgen\test_target_objective_generator.py, uv run --frozen --no-sync ruff format --check tests\unit\memory\test_sqlite_cancellation.py tests\unit\executor\promptgen\test_target_objective_generator.py, and uv run --frozen --no-sync ty check tests\unit\memory\test_sqlite_cancellation.py tests\unit\executor\promptgen\test_target_objective_generator.py: passed after the later main merge.
  • uv run --frozen --no-sync python -m build_scripts.validate_docs: passed. Commit-time pre-commit hooks also passed.

Updated doc/contributing/5_unit_tests.md. JupyText was not run because this PR changes no notebooks or JupyText sources. The complete GitHub OS/Python matrix has not been verified on the latest head.

Roman Lutz (romanlutz) and others added 3 commits October 6, 2026 12:50
Expire generator deadlines at the intended await, observe cancellation without watchdog-induced cancellation, and allow CI scheduling slack for coordination. Remove native spinner press-and-hold from single-step browser checks and document the async test patterns.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep upstream setup/send assertions alongside deterministic generator deadlines.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Allow CI scheduling slack before requesting cancellation and use non-cancelling completion waits while preserving rollback, cleanup, and exception assertions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants