Create community gets stuck after a brief network flap#3215
Open
holmesworcester wants to merge 1 commit into
Open
Create community gets stuck after a brief network flap#3215holmesworcester wants to merge 1 commit into
holmesworcester wants to merge 1 commit into
Conversation
The QSS auto-flow only re-emits QSS_HANDLE_SIGN_IN on QSS_CONNECTED.
When createCommunity returns false because the in-flight CREATE_COMMUNITY
ack rejected with 'socket has been disconnected' (or signInToCommunity
returns ERROR for the same kind of transient transport failure) and the
underlying websocket then settles back without a clean disconnect,
QSS_CONNECTED never re-fires while qssSetup stays false. Nothing
reschedules the create — the spinner sits forever even after the link
is fully healthy.
This breaks the loop with a single backoff-delayed retry scheduled
from inside _handleQssHandleSignIn:
- On createCommunity returning false or signInToCommunity returning
QSSOperationResult.ERROR, schedule one QSS_HANDLE_SIGN_IN re-emit
after _signInRetryDelayMs (starts at QSS_RECONNECT_DELAY_MS = 50 ms,
doubles up to QSS_RECONNECT_MAX_DELAY_MS = 60 s, matches the
existing _scheduleReconnect cadence so a sustained failure climbs
to the same cap instead of looping every 50 ms).
- On a SUCCESS / true result the timer is cleared and the delay
resets to its minimum.
- The timer is also cleared on pause() and close() to avoid leaks.
Reproducer (full toxiproxy chaos harness lives on the fork):
holmesworcester/quiet#14
branch: holmesworcester:poc/qss-flap-stalls-create
Caveat for the captcha-consumed case in particular: hCaptcha tokens
are single-use, so a same-socket retry of CREATE_COMMUNITY can't reuse
the original token. The retry path will call requestCaptchaVerification
again, which emits HCAPTCHA_CHALLENGE_REQUEST so the renderer can pop
the captcha modal. Whether the renderer actually re-shows the modal in
that state is a separate fix at a different layer; this PR only
addresses the backend-side wedge.
Adds a focused regression test in qss.service.spec.ts that mocks
createCommunity to return false once then true, calls connect() to
trigger the auto-flow, and waitForExpect()s a second invocation within
5 s. Without the fix the spy is called once; with the fix it is called
twice after the ~50 ms backoff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
940cb9f to
4c5d7ea
Compare
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Clicking Create community on a slightly flaky link (WiFi router rebooting, phone briefly switching cellular bands, captive-portal hiccup, etc.) leaves the spinner up indefinitely. The websocket eventually recovers, but the create flow has bailed out and the backend has no path to recover on its own — the only way forward is for the user to solve the hCaptcha challenge a second time, but the renderer never re-prompts.
This PR ships the smallest backend fix for the architectural shape behind it, plus a focused regression test in
qss.service.spec.ts. The full toxiproxy chaos repro lives on a fork branch (link below) so this PR doesn't depend on the stress harness infra that isn't in upstream7.1.0.Failure trace from the running backend
Code-path walkthrough (production code only, no test logic)
The auto-flow:
QSSService.connect()opens the websocket throughQSSClient.createSocketAndConnect. On success the client emitsQSS_CONNECTED(qss.client.ts:189).QSSService._handleQssConnected(qss.service.ts:199-202) emitsQSS_HANDLE_SIGN_IN.QSSService._handleQssHandleSignIn(qss.service.ts:217-254) takes the_signInMutexand, sinceqssSetupis false on a fresh community, callscreateCommunity->_createCommunityImpl(qss.service.ts:671-779)._createCommunityImpldoes, in order:requestCaptchaVerification(line 687) ->getCaptchaSiteKey->verifyCaptchaTokenGEN_PUB_KEYSsend-with-ack (line 717-721)CREATE_COMMUNITYsend-with-ack (line 762-766)What the flap does:
verifyCaptchaTokensucceeds. Inside it,qss.client.ts:319clears the renderer-side cached token:this.captchaService.hcaptchaToken = null. hCaptcha tokens are single-use, so this is correct in isolation.GEN_PUB_KEYSorCREATE_COMMUNITY) is in flight when the proxy drops. socket.io'semitWithAckrejects withError: socket has been disconnected.QSSClient.sendMessageswallows the rejection:qss.client.ts:269-271catches all errors, logs at error level, and returnsundefined._createCommunityImplreadsnull, logsFailed to create a community! Response was nullish, returnsfalse. The_signInMutex.runExclusivecallback resolves cleanly with no record that the in-flight failure was actually a disconnect.What happens on reconnect: only ONE
QSS_HANDLE_SIGN_INis emitted by_handleQssConnected. Since_createCommunityImplreturnsfalseonce and nothing reschedules, the auto-flow never tries again unless the websocket disconnects-and-reconnects a second time. Once the link settles cleanly there is no furtherQSS_CONNECTED, and the create is wedged.The fix (
qss.service.ts)Add a single backoff-delayed retry of
QSS_HANDLE_SIGN_INscheduled from inside the sign-in handler:_signInRetryTimer+_signInRetryDelayMs(matches the existing_reconnectQueueProcessorpattern, capped atQSS_RECONNECT_MAX_DELAY_MS = 60 s, doubles per failure viaQSS_RECONNECT_BACKOFF_FACTOR)._handleQssHandleSignIn: capture the result ofcreateCommunity/signInToCommunityand call_scheduleSignInRetry(...)onfalse/ERROR. Reset the backoff ontrue/SUCCESS.setTimeoutfires outside_signInMutex, so re-entering_handleQssHandleSignInworks.pause()andclose()clear the timer to avoid leaks during shutdown.This is the smallest backend-side fix that breaks the wedge. It also addresses the architectural twin tracked in #3216 — the same retry mechanism handles
signInToCommunityreturningERRORafter a flap.Caveat for the captcha-consumed case in particular
hCaptcha tokens are single-use, so a same-socket retry of
CREATE_COMMUNITYcannot reuse the original token. The retry path will callrequestCaptchaVerificationagain, which callscaptchaService.getToken(siteKey), which emitsHCAPTCHA_CHALLENGE_REQUESTso the renderer can pop the captcha modal. Whether the renderer actually re-shows the modal in that state is a separate fix at a different layer (renderer needs to listen toHCAPTCHA_CHALLENGE_REQUESTwhile a create is "in progress" in its own UI state). This PR only addresses the backend-side wedge — it ensures the backend keeps trying instead of going silent, and it ensures each retry emits a fresh challenge request so the renderer at least can re-prompt.The regression test (
qss.service.spec.ts)Without the fix,
createCommunityis called exactly once. With the fix it is called a second time after the ~50 ms initial backoff. Test exercises the production_handleQssHandleSignInauto-flow end-to-end (no internal mocking of the retry mechanism).Full toxiproxy chaos PoC
The deterministic toxiproxy stress repro that originally surfaced this bug lives on a fork branch — it depends on a stress harness (
packages/backend/src/nest/qss/stress/harness.ts, invariants helpers,jest.stress.config.js, etc.) that isn't in upstream7.1.0. To run it locally:holmesworcester:poc/qss-flap-stalls-createpackages/backend/src/nest/qss/stress/scenarios/flap-stalls-create.stress.spec.tsThe fork run boots a backend behind a TCP proxy and toggles the proxy at 5 Hz for 4 s while the QSSService auto-flow runs. After the link goes back to clean, it polls
qssSetupfor 90 s. On stock 7.1.0 the poll never succeeds; with the retry fix in this PR, the auto-flow recovers within the first backoff cycle.Test plan
qss.service.spec.tsregression test passes locally with the fix; fails (createCommunity called once) without it.tsc -p tsconfig.build.json --noEmitclean for changes in this PR.holmesworcester:poc/qss-flap-stalls-createagainst this branch'sqss.service.tsto confirm the 90 s post-flap deadline goes green.HCAPTCHA_CHALLENGE_REQUESTduring an in-progress create so the user is re-prompted automatically (out of scope here).