Skip to content

Fix IU HPC launcher lifecycle - #951

Merged
wangyinz merged 2 commits into
masterfrom
codex/issue-807-iu-hpc
Aug 25, 2026
Merged

wangyinz merged 2 commits into
masterfrom
codex/issue-807-iu-hpc

Conversation

@wangyinz

@wangyinz wangyinz commented Aug 12, 2026 •

Copy link
Copy Markdown
Member

Related to #807

Summary

  • initialize explicit/auto/mixed host and every process-handle state
  • execute exact tunnel/container/worker argv without shell metacharacters or shell=True
  • probe the configured MongoDB and Dask endpoints before workers/frontends
  • detect every owned-child early exit and startup timeout
  • terminate and wait for already-started owned children before reporting failure
  • retain a failed process handle when cleanup itself fails so shutdown can be retried
  • keep frontend launch nonblocking and retain its process handle

Renewed error-contract review

The issue text changed all startup failures to MsPASSError(Invalid), but this class is an infrastructure launcher and its baseline public behavior uses ordinary Python runtime/configuration exceptions. Commit 60be0795 restores that boundary:

  • malformed timeout/poll settings raise ValueError before any child starts
  • owned-child early exit/readiness timeout raises RuntimeError
  • process-creation/system exceptions are re-raised with their original identity after cleanup
  • if cleanup fails, that failure is explicit and the failed handle is retained rather than silently discarded

The long launch and shutdown docstrings now document these boundaries.

Validation

  • focused launcher lifecycle contract: 29/29 passed
  • added original-exception identity and cleanup-retry-handle tests
  • Python 3.12/3.13 py_compile and git diff --check: passed
  • the new test file is Black 26.5.1 clean; the pre-existing launcher file has legacy whole-file formatting, so unrelated reformatting was deliberately excluded

This remains draft pending fresh CI and human infrastructure/API review.

@codecov

codecov Bot commented Aug 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.35%. Comparing base (482218e) to head (fbe477a).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #951   +/-   ##
=======================================
  Coverage   74.35%   74.35%           
=======================================
  Files         183      183           
  Lines       36516    36516           
  Branches     2540     2540           
=======================================
  Hits        27151    27151           
  Misses       8455     8455           
  Partials      910      910           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wangyinz
wangyinz marked this pull request as draft August 14, 2026 06:01
@wangyinz
wangyinz marked this pull request as ready for review August 14, 2026 15:46
@wangyinz
wangyinz marked this pull request as draft August 25, 2026 16:17
@wangyinz

Copy link
Copy Markdown
Member Author

Renewed review found three blockers before merge:

  1. Drop the test_workflow.py commit; that lifecycle fix was already superseded by Isolate the workflow test Dask client lifecycle #1004/Fix sliding-window pipeline result and resource contracts #885 on master.
  2. Normalize an auto-discovered Dask host to the same complete endpoint used by workers (default tcp://HOST:8786) before the readiness probe. A bare host can hang beyond the advertised 2-second boundary.
  3. Do not pipe long-lived child stdout/stderr unless they are drained. Inheriting the batch job output is the smallest safe default.

Rebase and merge this PR before #952 because both edit launcher.py. No maintainer API decision remains for #951.

@wangyinz
wangyinz force-pushed the codex/issue-807-iu-hpc branch from 60be079 to fbe477a Compare August 25, 2026 16:44
@wangyinz

Copy link
Copy Markdown
Member Author

Implemented the renewed review fixes on current master:

  • dropped the superseded test_workflow.py lifecycle commit
  • normalized auto Dask readiness addresses to tcp://HOST:8786 (or DASK_SCHEDULER_PORT)
  • removed undrained pipes from long-lived owned children
  • parsed configured runtime arguments with shlex

Final local validation: 30 focused launcher tests, py_compile, and git diff --check passed. The PR remains Draft until this final head passes CI.

@wangyinz
wangyinz marked this pull request as ready for review August 25, 2026 17:11
@wangyinz
wangyinz merged commit 7d00c61 into master Aug 25, 2026
22 checks passed
@wangyinz
wangyinz deleted the codex/issue-807-iu-hpc branch August 25, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant