Skip to content

feat(qss-stress): consolidate harness — precise faults, parallel proxies, retry#15

Closed
holmesworcester wants to merge 1 commit into
7.1.0from
feat/qss-stress-consolidation
Closed

feat(qss-stress): consolidate harness — precise faults, parallel proxies, retry#15
holmesworcester wants to merge 1 commit into
7.1.0from
feat/qss-stress-consolidation

Conversation

@holmesworcester

Copy link
Copy Markdown
Owner

Summary

Wraps up the QSS stress harness with the last pieces from the investigation work plus the parallelism / robustness improvements that weren't part of the earlier per-peer-chaos / leave-switch / pause-primitive merges.

What's in

Infra:

  • precise-faults.tsinstallWireFaultHooks() monkey-patches qssClient.sendMessage to inject precise per-message faults (drop / delay / error / busy-wait pause) keyed by WebsocketEvents type. Helpers: scheduleOutageAfter, oneShot, busyWaitPause. Use it when network-level toxiproxy chaos is too coarse and the bug needs message-type granularity.
  • harness.ts — per-test proxy isolation. Every bootQssHarness() / bootMemberHarness() call allocates a unique proxy name and a free local port, and deletes the proxy on shutdown. Lets multiple stress runs execute in parallel under jest's --maxWorkers without colliding on a shared qss proxy.
  • toxiproxy.ts — retry-with-backoff for transient 503/timeout/ECONNRESET on the toxiproxy admin API. The admin API is single-threaded; under concurrent load it returns transient errors that are safe to retry. Also splits rawRequest from request and surfaces HTTP status on errors.

Scenarios:

  • scenarios/flap-stalls-create.stress.spec.ts — rapid proxy enable/disable flaps during preCreate and duringAuthSync. Surfaces stalls in the sign-in / auth-conn state machines under churn.
  • scenarios/wire-fault-acks.stress.spec.ts — per-message ack-handling sweep using installWireFaultHooks. Demonstrates wire-fault hook usage.

Scripts:

  • Root: start:qss:toxi / stop:qss:toxi convenience wrappers around docker compose for the toxiproxy sidecar.
  • Backend: test-stress-qss script for running the dedicated jest config. testPathIgnorePatterns excludes stress/ from default test/test-ci so CI is unaffected.

Test plan

  • docker compose up toxiproxy sidecar
  • npm run start:qss (existing)
  • cd packages/backend && npm run test-stress-qss — verify the suite runs end-to-end against a real QSS server
  • Confirm npm run test/test-ci does not pick up *.stress.spec.ts (excluded via testPathIgnorePatterns)

…ies, retry

Wraps up the QSS stress harness with the last pieces from the
investigation work and the parallelism / robustness improvements that
weren't part of the earlier per-peer-chaos / leave-switch / pause-primitive
merges.

Infra additions:
- precise-faults.ts — installWireFaultHooks() monkey-patches
  qssClient.sendMessage to inject precise per-message faults (drop, delay,
  error, busy-wait pause) keyed by WebsocketEvents type. Helpers:
  scheduleOutageAfter, oneShot, busyWaitPause. Use it when network-level
  toxiproxy chaos is too coarse and the bug needs message-type granularity.
- harness.ts — per-test proxy isolation: every bootQssHarness() /
  bootMemberHarness() call allocates a unique proxy name and a free local
  port, and deletes the proxy on shutdown. Lets multiple stress runs
  execute in parallel under jest's --maxWorkers without colliding on a
  shared 'qss' proxy.
- toxiproxy.ts — retry-with-backoff for transient 503/timeout/ECONNRESET
  on the toxiproxy admin API. The admin API is single-threaded; under
  concurrent load it returns transient errors that are safe to retry.
  Also splits rawRequest from request and surfaces HTTP status on errors.

Scenarios:
- scenarios/flap-stalls-create.stress.spec.ts — rapid proxy enable/disable
  flaps during preCreate and duringAuthSync. Surfaces stalls in the
  sign-in / auth-conn state machines under churn.
- scenarios/wire-fault-acks.stress.spec.ts — per-message ack-handling
  sweep using installWireFaultHooks. Demonstrates wire-fault hook usage.

Scripts:
- Root: start:qss:toxi / stop:qss:toxi convenience wrappers around
  docker compose for the toxiproxy sidecar.
- Backend: test-stress-qss script for running the dedicated jest config.
  testPathIgnorePatterns excludes stress/ from default test/test-ci so
  CI is unaffected.
@holmesworcester

Copy link
Copy Markdown
Owner Author

Superseded by TryQuiet/quiet#3219, which lands just the toxiproxy tooling (harness, precise-faults, toxiproxy client, jest config, scripts) on 7.1.0. The scenario specs / fuzz catalog / invariants / FINDINGS will land separately. Closing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant