Skip to content

Create community gets stuck after a brief network flap#3215

Open
holmesworcester wants to merge 1 commit into
7.1.0from
poc/qss-flap-stalls-create-upstream
Open

Create community gets stuck after a brief network flap#3215
holmesworcester wants to merge 1 commit into
7.1.0from
poc/qss-flap-stalls-create-upstream

Conversation

@holmesworcester

@holmesworcester holmesworcester commented May 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Clicking Create community on a slightly flaky link (WiFi router rebooting, phone briefly switching cellular bands, captive-portal hiccup, etc.) leaves the spinner up indefinitely. The websocket eventually recovers, but the create flow has bailed out and the backend has no path to recover on its own — the only way forward is for the user to solve the hCaptcha challenge a second time, but the renderer never re-prompts.

This PR ships the smallest backend fix for the architectural shape behind it, plus a focused regression test in qss.service.spec.ts. The full toxiproxy chaos repro lives on a fork branch (link below) so this PR doesn't depend on the stress harness infra that isn't in upstream 7.1.0.

Failure trace from the running backend

QSS connected
GET_CAPTCHA_SITE_KEY ack    -> ok
VERIFY_CAPTCHA       ack    -> ok                   <-- token consumed
GEN_PUB_KEYS         ack    -> ok
CREATE_COMMUNITY     emit   -> socket disconnects mid-flight
ERROR Error while sending message to QSS Error: socket has been disconnected
ERROR Failed to create a community! Response was nullish

(reconnect, captchaVerified is now false)
GET_CAPTCHA_SITE_KEY ack    -> ok
INFO  Requesting hCaptcha token from renderer process
WARN  Failed to obtain hCaptcha token for verification     <-- 30 s renderer timeout
WARN  Can't create community on QSS because captcha verification failed
(loop forever)

Code-path walkthrough (production code only, no test logic)

The auto-flow:

  1. QSSService.connect() opens the websocket through QSSClient.createSocketAndConnect. On success the client emits QSS_CONNECTED (qss.client.ts:189).
  2. QSSService._handleQssConnected (qss.service.ts:199-202) emits QSS_HANDLE_SIGN_IN.
  3. QSSService._handleQssHandleSignIn (qss.service.ts:217-254) takes the _signInMutex and, since qssSetup is false on a fresh community, calls createCommunity -> _createCommunityImpl (qss.service.ts:671-779).
  4. _createCommunityImpl does, in order:
    • requestCaptchaVerification (line 687) -> getCaptchaSiteKey -> verifyCaptchaToken
    • GEN_PUB_KEYS send-with-ack (line 717-721)
    • CREATE_COMMUNITY send-with-ack (line 762-766)

What the flap does:

  • verifyCaptchaToken succeeds. Inside it, qss.client.ts:319 clears the renderer-side cached token: this.captchaService.hcaptchaToken = null. hCaptcha tokens are single-use, so this is correct in isolation.
  • The next message (GEN_PUB_KEYS or CREATE_COMMUNITY) is in flight when the proxy drops. socket.io's emitWithAck rejects with Error: socket has been disconnected.
  • QSSClient.sendMessage swallows the rejection: qss.client.ts:269-271 catches all errors, logs at error level, and returns undefined.
  • _createCommunityImpl reads null, logs Failed to create a community! Response was nullish, returns false. The _signInMutex.runExclusive callback resolves cleanly with no record that the in-flight failure was actually a disconnect.

What happens on reconnect: only ONE QSS_HANDLE_SIGN_IN is emitted by _handleQssConnected. Since _createCommunityImpl returns false once and nothing reschedules, the auto-flow never tries again unless the websocket disconnects-and-reconnects a second time. Once the link settles cleanly there is no further QSS_CONNECTED, and the create is wedged.

The fix (qss.service.ts)

Add a single backoff-delayed retry of QSS_HANDLE_SIGN_IN scheduled from inside the sign-in handler:

  • New _signInRetryTimer + _signInRetryDelayMs (matches the existing _reconnectQueueProcessor pattern, capped at QSS_RECONNECT_MAX_DELAY_MS = 60 s, doubles per failure via QSS_RECONNECT_BACKOFF_FACTOR).
  • In _handleQssHandleSignIn: capture the result of createCommunity / signInToCommunity and call _scheduleSignInRetry(...) on false / ERROR. Reset the backoff on true / SUCCESS.
  • The retry's setTimeout fires outside _signInMutex, so re-entering _handleQssHandleSignIn works.
  • pause() and close() clear the timer to avoid leaks during shutdown.

This is the smallest backend-side fix that breaks the wedge. It also addresses the architectural twin tracked in #3216 — the same retry mechanism handles signInToCommunity returning ERROR after a flap.

Caveat for the captcha-consumed case in particular

hCaptcha tokens are single-use, so a same-socket retry of CREATE_COMMUNITY cannot reuse the original token. The retry path will call requestCaptchaVerification again, which calls captchaService.getToken(siteKey), which emits HCAPTCHA_CHALLENGE_REQUEST so the renderer can pop the captcha modal. Whether the renderer actually re-shows the modal in that state is a separate fix at a different layer (renderer needs to listen to HCAPTCHA_CHALLENGE_REQUEST while a create is "in progress" in its own UI state). This PR only addresses the backend-side wedge — it ensures the backend keeps trying instead of going silent, and it ensures each retry emits a fresh challenge request so the renderer at least can re-prompt.

The regression test (qss.service.spec.ts)

describe('QSS_HANDLE_SIGN_IN retry on createCommunity false (regression)', () => {
  it('retries createCommunity once after a single false result', async () => {
    // qssSetup=false so the create branch runs
    await initCommunity({ qssEnabled: true, qssSetup: false })
    mockedAllowed.mockReturnValue(true)
    const createSpy = jest.spyOn(qssService, 'createCommunity')
      .mockResolvedValueOnce(false)  // first attempt fails
      .mockResolvedValueOnce(true)   // retry succeeds
    await qssService.connect('ws://localhost:3000')
    await waitForExpect(() => {
      expect(createSpy).toHaveBeenCalledTimes(2)        // <-- without fix, called 1×
    }, 5000)
  })
})

Without the fix, createCommunity is called exactly once. With the fix it is called a second time after the ~50 ms initial backoff. Test exercises the production _handleQssHandleSignIn auto-flow end-to-end (no internal mocking of the retry mechanism).

Full toxiproxy chaos PoC

The deterministic toxiproxy stress repro that originally surfaced this bug lives on a fork branch — it depends on a stress harness (packages/backend/src/nest/qss/stress/harness.ts, invariants helpers, jest.stress.config.js, etc.) that isn't in upstream 7.1.0. To run it locally:

The fork run boots a backend behind a TCP proxy and toggles the proxy at 5 Hz for 4 s while the QSSService auto-flow runs. After the link goes back to clean, it polls qssSetup for 90 s. On stock 7.1.0 the poll never succeeds; with the retry fix in this PR, the auto-flow recovers within the first backoff cycle.

Test plan

  • qss.service.spec.ts regression test passes locally with the fix; fails (createCommunity called once) without it.
  • tsc -p tsconfig.build.json --noEmit clean for changes in this PR.
  • Maintainer: run the toxiproxy PoC on holmesworcester:poc/qss-flap-stalls-create against this branch's qss.service.ts to confirm the 90 s post-flap deadline goes green.
  • Maintainer follow-up: renderer-side handling of HCAPTCHA_CHALLENGE_REQUEST during an in-progress create so the user is re-prompted automatically (out of scope here).

The QSS auto-flow only re-emits QSS_HANDLE_SIGN_IN on QSS_CONNECTED.
When createCommunity returns false because the in-flight CREATE_COMMUNITY
ack rejected with 'socket has been disconnected' (or signInToCommunity
returns ERROR for the same kind of transient transport failure) and the
underlying websocket then settles back without a clean disconnect,
QSS_CONNECTED never re-fires while qssSetup stays false. Nothing
reschedules the create — the spinner sits forever even after the link
is fully healthy.

This breaks the loop with a single backoff-delayed retry scheduled
from inside _handleQssHandleSignIn:

  - On createCommunity returning false or signInToCommunity returning
    QSSOperationResult.ERROR, schedule one QSS_HANDLE_SIGN_IN re-emit
    after _signInRetryDelayMs (starts at QSS_RECONNECT_DELAY_MS = 50 ms,
    doubles up to QSS_RECONNECT_MAX_DELAY_MS = 60 s, matches the
    existing _scheduleReconnect cadence so a sustained failure climbs
    to the same cap instead of looping every 50 ms).
  - On a SUCCESS / true result the timer is cleared and the delay
    resets to its minimum.
  - The timer is also cleared on pause() and close() to avoid leaks.

Reproducer (full toxiproxy chaos harness lives on the fork):
  holmesworcester/quiet#14
  branch: holmesworcester:poc/qss-flap-stalls-create

Caveat for the captcha-consumed case in particular: hCaptcha tokens
are single-use, so a same-socket retry of CREATE_COMMUNITY can't reuse
the original token. The retry path will call requestCaptchaVerification
again, which emits HCAPTCHA_CHALLENGE_REQUEST so the renderer can pop
the captcha modal. Whether the renderer actually re-shows the modal in
that state is a separate fix at a different layer; this PR only
addresses the backend-side wedge.

Adds a focused regression test in qss.service.spec.ts that mocks
createCommunity to return false once then true, calls connect() to
trigger the auto-flow, and waitForExpect()s a second invocation within
5 s. Without the fix the spy is called once; with the fix it is called
twice after the ~50 ms backoff.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant