Skip to content

Simplex Desktop crash recovery and diagnostics - #1405

Merged
Wizdave97 merged 14 commits into
mainfrom
simplex-release-crash-testing
Oct 22, 2026
Merged

Wizdave97 merged 14 commits into
mainfrom
simplex-release-crash-testing

Conversation

@ddboy19912

@ddboy19912 ddboy19912 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Simplex Desktop recovers a crashed dashboard without quitting the shell or restarting the separate solver. It reloads automatically once, then offers an explicit retry after each later crash, including a crash during a requested reload. Initial navigation failures allow up to one second for a renderer-exit event; genuine load errors remain fatal.

OS secure-storage checks return unavailable after five seconds so password login remains accessible and encrypted. The locked login screen refreshes unavailable storage sequentially, restoring passkey access when the native check completes while preserving the selected password form.

The PR also bounds desktop socket requests, detects unexpected solver disappearance on all platforms, and adds scrubbed, rotating local diagnostics. Main-process errors and solver exits share one desktop log. Renderer exits are logged locally; the existing Sentry child-process integration handles remote crash reports without a duplicate exception event. Recovery failures still use desktop error reporting.

Sentry setup, opt-out and scrubbing already landed in #1399. Its documentation remains unchanged. This PR's behavior is documented in docs/ai/changelog/2026-10-14-simplex-desktop-crash-recovery.md.

Validation:

  • Desktop unit/integration: 300 passed, one platform skip. UI: 134 passed.
  • Native macOS Electron: 21 passed, one platform skip; final login/recovery checks also passed after the last UI adjustment.
  • Desktop/UI type checks, lint and builds passed.
  • Regression coverage includes a crash before a manual reload rejects, both retry choices, late storage availability, request cancellation, password-form preservation, and shared diagnostics with one Sentry recovery-error event.
  • Desktop CI passed on macOS, Windows, Linux x64 and Linux ARM64: https://github.com/polytope-labs/hyperbridge/actions/runs/39270296646. The normal job limit is 20 minutes and default page-action timeout is 30 seconds.
  • The separate SDK workflow failed before reaching the Simplex suites: bid submission/retraction exceeded the 20-second transaction-pool deadline in unchanged intentsCoprocessor.test.ts: https://github.com/polytope-labs/hyperbridge/actions/runs/39270296681.
  • React Doctor flagged the recursive refresh timer as missing cleanup. The effect aborts its request and clears its timer; regression tests verify cancellation before polling and after an in-flight request completes. No suppression was added.

The reported Windows password-entry hang has not been reproduced on the affected PC; these verified defects are not established as its cause.

The web UI (@sentry/react) and the desktop main process (@sentry/electron)
report errors when CI bakes a DSN into the build. Events are scrubbed of
recovery phrases, keys, long tokens and provider URL paths before sending.
Operators opt out with SIMPLEX_ERROR_REPORTING=off: /health tells the UI
to drop events, and the desktop skips Sentry and keeps it out of the
renderer CSP. UI source maps are uploaded in CI and deleted from dist.
Renderer, GPU and utility process crashes and OOMs are sent as events.
An unexpected solver stop is reported where the desktop already notifies
the operator. A running-session marker in userData reports a main
process that ended without a clean quit on the next launch, ignoring
sessions that an OS shutdown or restart ended.
Gate SDK initialization because session envelopes bypass beforeSend. Scrub mixed-case and structured recovery phrases, absolute file paths, and cyclic data. Enable source maps in an entry module before loading the desktop main bundle.
@ddboy19912 ddboy19912 changed the title Simplex release crash testing Simplex Desktop crash recovery and diagnostics Oct 15, 2026

@Wizdave97 Wizdave97 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: needs work before it leaves draft. I reviewed the net diff against main.

Scope: the Sentry work described in the PR (opt-out, scrubbing, no minidumps) is already on main via #1399. This branch's net diff doesn't change the Sentry setup, so the description should cover only what this diff adds. The only new data sent remotely is the renderer-exit event in (3).

What holds up:

  • There's no reload loop: one automatic reload, then every later reload waits for the user.
  • The 1 s wait for the crash event works for both event orders.
  • socketRequest's hard deadline handles interrupted and oversized responses.
  • Removal of the Windows named pipe now raises the alert.
  • The storage-check timeout never falls back to plaintext: wrapDeviceKey returns undefined and the existing deviceKey is kept.

Must fix

  1. After the user clicks "Reload dashboard", the next crash shows no prompt on macOS and Windows (renderer-recovery.ts:48-59).

    • Sequence: crash 1 reloads automatically and crash 2 shows the prompt. The user clicks Reload, and crash 3 happens while that reload is still loading.
    • this.prompting is still set at that point, so crashed() only awaits it, and the window is left crashed and blank.
    • Linux delivers the crash event after loadURL rejects, so Linux does prompt again. A small simulation using the macOS/Windows order ends at {prompts: 1, reloads: 2, crashedNow: true}.
    • The e2e test always answers "Later", so this path never runs in tests.
    • Suggestion: clear this.prompting after prompt() resolves and before show(), or prompt again if a crash landed during the retry. Add a unit test where the crash fires while the prompted reload is pending.
  2. Remove the investigation scaffolding.

    • In .github/workflows/test-simplex-desktop.yml: the test_published_windows job pinned to v0.17.4, its checksum, and SIMPLEX_TEST_RELOADS=1000.
    • In scripts/e2e/desktop.e2e.mjs: crashAuditEvents, which is collected but never asserted, the debug JSON console.log calls, and the blocks indented with spaces.
    • Reconsider the global page.setDefaultTimeout(120_000) and the job timeout going from 20 to 30 minutes, since both can hide hangs.
  3. Each renderer crash appears to send two Sentry events (main.ts:450). childProcessIntegration already reports renderer crashed, oom and abnormal-exit exits, and reportDesktopError adds a second captureException. Send crash notifications to the local diagnostics log only, and keep Sentry for reload failures.

Should fix

  1. A storage check that is slow but succeeds leaves the login screen stuck on "unavailable" (device-key-store.ts:30).

    • Suppose the check takes about 8 s, for example while a keychain or keyring prompt waits for the user. backgroundKey() returns false at 5 s, and state() keeps reporting passkeyAvailable: false from the cached promise.
    • The UI fetches /api/desktop/security once, on mount. So a passkey-only profile only sees the recovery-code form, where before this PR it would have waited and then offered the passkey.
    • Suggestion: poll again while secureDeviceStorageAvailable is false, or push a refresh (and retry backgroundKey) when the check completes.
  2. Many tiny desktop diagnostic files can push out the important ones (daemon.ts:107).

    • Every spawnDaemon call creates a new createDiagnosticLog(..., "desktop"), and each one opens its own file.
    • All desktop loggers share one 5-file limit, so files holding a single solver-exit line can delete the main process's error/uncaught file.
    • Suggestion: reuse the single desktop logger created in error-reporting.ts.

Docs:

  • The PR renames #1399's merged note docs/ai/changelog/2026-10-14-simplex-sentry-error-reporting.md to …-simplex-desktop-crash-recovery.md and folds its own content into it. Please restore the Sentry note and put this PR's content in a note of its own.
  • The note's ## Validation section belongs in the PR description.
  • Drop the history and incident wording, e.g. "no longer leaves supervision permanently waiting" and "cannot recover evidence from an older installed build".

Also rebase: the branch is 2 commits behind main. #1407 touches daemon.ts, though it merges cleanly.

Already on main, not part of this diff, but worth a follow-up:

  • Release builds report errors by default. The only opt-out is the SIMPLEX_ERROR_REPORTING environment variable, which is hard to set when the app is launched from the GUI, and there is no in-app toggle.
  • The scrubber is regex-based, so tokens under 32 characters that aren't inside a URL (e.g. pim_… keys, passwords) pass through unscrubbed.

@ddboy19912
ddboy19912 marked this pull request as ready for review October 19, 2026 16:54
@ddboy19912
ddboy19912 requested a review from Wizdave97 October 19, 2026 16:54

@Wizdave97 Wizdave97 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. All round-1 items are addressed:

  • The retry prompt shows again after a manual reload.
  • Each crash sends one Sentry event.
  • A slow storage check now refreshes the login screen.
  • The desktop diagnostic log is shared.
  • The investigation scaffolding is removed.
  • The #1399 note is restored.

Desktop E2E passes on all four platforms.

Non-blocking follow-ups:

  • Prettier. desktop.e2e.mjs has about 10 unrelated blocks reformatted in a way that fails the repo's Prettier config. Running pnpm --dir sdk/packages/simplex-desktop format, or reverting those hunks, fixes it.
  • Redundant timeout. page.setDefaultTimeout(30_000) (desktop.e2e.mjs:110) is Playwright's default and can go.
  • Endless polling. When secure storage is permanently unavailable, the locked screen polls every 2 s forever. A storageCheckPending flag in the state would let it stop.

@Wizdave97
Wizdave97 merged commit a1dfc61 into main Oct 22, 2026
26 of 30 checks passed
@Wizdave97
Wizdave97 deleted the simplex-release-crash-testing branch October 22, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants