listCodexModels spawns new codex app-server on every request, causing intermittent 120s timeouts

[creeep123]

Problem

GET /api/sessions/{id}/codex-models intermittently returns 500 after 120 seconds, which gets wrapped as a 502 Bad Gateway by Cloudflare Tunnel.

Root Cause

In cli/src/modules/common/codexModels.ts, listCodexModels() creates a new CodexAppServerClient on every call, which spawns a fresh codex app-server child process:

export async function listCodexModels(includeHidden: boolean = false): Promise<CodexModelSummary[]> {
    const client = new CodexAppServerClient();
    try {
        await client.connect();        // spawns "codex app-server" process
        await client.initialize(...);   // 30s timeout
        const response = await client.listModels({ includeHidden }); // 30s timeout
        ...
    } finally {
        await client.disconnect();     // kills the process
    }
}

The hub-side RPC timeout is MODEL_LIST_RPC_TIMEOUT_MS = 120_000 (120s). When the spawned codex app-server is slow to respond (e.g., OpenAI token refresh stalls), the RPC times out at 120s and returns 500.

Evidence

From hub.log, out of ~168 codex-models requests:

• 163 (97%) succeeded in 1-5 seconds
• 5 (3%) timed out at exactly 120s → 500

From runner.log, the codex app-server processes spawned for model listing consistently exit within ~1 second:

[09:21:39.748] List Codex models request
[09:21:39.748] [CodexAppServer] Connected
[09:21:40.147] Codex app-server exited (code=0, signal=null)
[09:21:40.157] [CodexAppServer] Disconnected

The 1-second exit suggests the app-server sometimes fails silently (exits before completing listModels), triggering the full 120s RPC timeout on the hub side.

Environment

• hapi: 0.19.0
• codex-cli: 0.136.0
• OS: Ubuntu 24.04 (Linux x86_64)
• codex auth mode: chatgpt (OAuth token with refresh)

Suggested Fix

1. Cache model list on the runner/machine level (models rarely change, cache for 5-10 minutes)
2. Reuse a persistent app-server instead of spawn-per-request
3. Reduce RPC timeout — 120s is excessive for a model list that normally takes <5s
4. Add faster failure detection — if the app-server exits early, return an error immediately instead of waiting for the full RPC timeout

[github-actions]

Confirmed the main code path in current source.

GET /api/sessions/:id/codex-models calls the hub sync engine, which sends listCodexModels over RPC with a 120s Socket.IO ACK timeout. On the CLI side, that RPC handler calls listCodexModels(), and listCodexModels() constructs a new CodexAppServerClient for each request. connect() spawns codex app-server; initialize() and model/list each use 30s app-server request timeouts.

One detail in the proposed root cause does not match the current client code: when the codex app-server child exits, CodexAppServerClient calls rejectAllPending(...), so an early child exit should normally return an RPC error response instead of waiting for the full 120s hub ACK timeout. The 120s timeout points to the hub not receiving an RPC ACK for that request, or the CLI handler not completing, but the exact reason is not visible from source alone.

Relevant code:

• hub/src/web/routes/sessions.ts:567 - session Codex model-list endpoint
• hub/src/web/routes/sessions.ts:587 - endpoint calls engine.listCodexModelsForSession(...)
• hub/src/sync/rpcGateway.ts:24 - model-list RPC timeout is 120_000
• hub/src/sync/rpcGateway.ts:237 - session Codex model listing uses that timeout
• hub/src/sync/rpcGateway.ts:290 - timeout is applied to Socket.IO emitWithAck(...)
• cli/src/modules/common/handlers/codexModels.ts:12 - CLI registers the listCodexModels RPC handler
• cli/src/modules/common/codexModels.ts:53 - listCodexModels() creates a new client per call
• cli/src/codex/codexAppServerClient.ts:94 - client spawns codex app-server
• cli/src/codex/codexAppServerClient.ts:150 - initialize timeout is 30s
• cli/src/codex/codexAppServerClient.ts:156 - model/list timeout is 30s
• cli/src/codex/codexAppServerClient.ts:117 - child exit handler rejects pending requests

Need more info:

• One matching hub log entry for a 120s timeout, including the exact error text.
• Matching runner log lines for the same timestamp, including any [CodexAppServer][stderr] output.
• Whether the affected request was /api/sessions/:id/codex-models or /api/machines/:id/codex-models.

---

HAPI Bot

[creeep123]

Additional debugging info as requested

1. Hub log — 120s timeout entries

The hub log only records <-- GET /api/sessions/:id/codex-models (incoming) and --> GET /api/sessions/:id/codex-models 500 120s (outgoing). There is no intermediate error text logged between the request and the 120s timeout response — the hub silently waits the full RPC ACK timeout and returns 500.

Sample (line 45508):

--> GET /api/sessions/c6d0eca0-5be5-4061-b3d8-aea7bc107298/codex-models 500 120s

No additional error/debug context is logged by the hub for these failures.

2. Runner log — [CodexAppServer][stderr] output (root cause confirmed)

The runner log does contain the smoking gun. Two entries showing the codex app-server failing with 401 Unauthorized due to an invalidated token:

2025-06-05T23:09:28 (codex-cli 0.136.0):

[23:09:28.597] List Codex models request
[23:09:28.599] [CodexAppServer] Connected
[23:09:29.068] [CodexAppServer][stderr] 2026-06-05T15:09:29.068327Z ERROR codex_models_manager::manager: failed to refresh available models: unexpected status 401 Unauthorized: Your authentication token has been invalidated. Please try signing in again., url: https://chatgpt.com/backend-api/codex/models?client_version=0.136.0, cf-ray: a07019401d2f06e1-IAD, request id: 03601f1d-b792-473b-9b78-cec437c49786, auth error: 401, auth error code: token_invalidated
[23:09:29.109] Codex app-server exited (code=0, signal=null)
[23:09:29.123] [CodexAppServer] Disconnected

2026-06-06T13:07:43 (codex-cli 0.137.0, after fresh codex login --device-auth):

[13:07:43.872] List Codex models request
[13:07:43.876] [CodexAppServer] Connected
[13:07:44.374] [CodexAppServer][stderr] 2026-06-06T05:07:44.373772Z ERROR codex_models_manager::manager: failed to refresh available models: unexpected status 401 Unauthorized: Your authentication token has been invalidated. Please try signing in again., url: https://chatgpt.com/backend-api/codex/models?client_version=0.137.0, cf-ray: a074e529cc8c2ac2-IAD, request id: 2531590d-a8f0-4fe4-9fc1-bc6a5bd1f635, auth error: 401, auth error code: token_invalidated
[13:07:44.431] Codex app-server exited (code=0, signal=null)
[13:07:44.444] [CodexAppServer] Disconnected

Key observation: The second failure happened after a fresh codex login --device-auth. The login was performed as root but the hapi runner runs as user moses — so the new credentials were saved to /root/.codex/auth.json while the runner reads /home/moses/.codex/auth.json (which still had the old invalidated token). This is a separate ops issue on our side, but the code behavior is worth noting.

3. Endpoint type

All 120s timeouts are on /api/sessions/:id/codex-models. The /api/machines/:id/codex-models endpoint always succeeds (200, 400ms–3s).

4. Updated statistics

Out of 361 total codex-models requests in hub.log:

• 260 (72%) — 200 (400ms–3s)
• 101 (28%) — 500 at exactly 120s
• ~10 — 500 at 0–6ms (fast failures, likely no runner connected)

5. Refined root cause hypothesis

Given the [CodexAppServer][stderr] evidence, here is the updated sequence:

1. Hub receives GET /api/sessions/:id/codex-models and sends listCodexModels RPC to runner (120s ACK timeout)
2. Runner creates CodexAppServerClient, spawns codex app-server child process
3. codex app-server attempts to refresh models from OpenAI API → receives 401 Unauthorized (token invalidated)
4. codex app-server exits with code 0 (clean exit, not a crash) in ~400ms
5. CodexAppServerClient detects child exit and calls rejectAllPending(...) — but this rejection may not propagate correctly back through the RPC layer to the hub's emitWithAck
6. Hub never receives the RPC ACK, waits full 120s, then returns 500

The bot's point about rejectAllPending(...) is valid — in theory the rejection should propagate. The fact that it still hits 120s suggests either:

• The rejection happens but the RPC transport (Socket.IO) doesn't deliver it as an ACK response, or
• There is a race condition between the child exit handler and the RPC response channel, or
• The listCodexModels() handler catches/rejects but the runner's RPC layer swallows the error before sending it back

A fix should ensure that if the codex app-server exits before completing listModels(), the RPC handler sends an immediate error ACK back to the hub instead of letting it silently hang.

6. Sessions most affected

Timeouts are concentrated in a few long-lived sessions (which were active during the auth-invalidated period):

• c951136a — 19 timeouts
• c6d0eca0 — 19 timeouts (also has 45 generated-images 404s)
• 5ad9e20e — 11 timeouts
• 29bda2aa — 10 timeouts