[P2][IU launcher] Repair HPC host, command, readiness, and process-state contracts #807

Baseline

Verified on master commit 61b06b965265a1327b84a8c6b6d7de491d9db97e.

Problem

All-explicit hosts leave attributes unset; scheduler assignment is misspelled; tunnel executes stale argv; scheduler/worker endpoints are wrong or incomplete; remote env lacks a delimiter and contains literal &; status treats exit 0 as running and polls absent workers; frontend omits endpoints and blocks on communicate.

Impact

Common HPC configurations fail, launch against wrong services, misreport stopped processes, or block interactive use.

Acceptance criteria

Initialize primary_node, database_host, scheduler_host, worker_hosts, and every process handle for all-explicit, all-auto, and mixed host configurations. Execute the newly constructed tunnel/container/worker argv exactly, pass configured Mongo and Dask endpoints in complete comma-delimited environment values, and use neither literal & nor shell=True. Only process.poll() is None is running; skip absent optional worker/frontend handles without polling or treating them as failures. Use MSPASS_STARTUP_TIMEOUT_SECONDS default 120 and MSPASS_STARTUP_POLL_SECONDS default 2. Before workers or frontend, require Mongo ping on the configured endpoint and a Dask probe that creates distributed.Client(configured_endpoint, timeout='2s'), calls scheduler_info(), and closes it. On owned-child early exit or timeout, start no worker/frontend, terminate and wait for all already-started owned children, and raise MsPASSError with ErrorSeverity.Invalid. Interactive launch includes both endpoints, returns immediately without communicate(), and retains its process handle. Limit this PR to HPCClusterLauncher and tests.

Required tests

Mock exact argv/environment for all-explicit, all-auto, mixed, tunnel, primary-worker, remote-worker, batch, and interactive paths, asserting no shell metacharacter or shell=True. Cover status for None, 0, nonzero, and absent optional handles; default/overridden timeout/poll settings; delayed Mongo/Dask readiness; each child early exit and timeout; Dask close; no later launch; owned-child terminate/wait; exact Invalid exception; nonblocking frontend and retained handle; YAML parse and py_compile.

Related

Merged #640 introduced the prototype but does not repair it.