Skip to content

Commit 2ed2fe6

Browse files
emdevelopaemdevelopa
authored
Merge pull request #1602 from noevidence1017/feature/be-websocket-relay-retry-telemetry-log-retention-validation
feat(backend): WebSocket relay retry and telemetry, payment processor telemetry suites, log retention validation
2 parents d73b20d + e1838ed commit 2ed2fe6

15 files changed

Lines changed: 2266 additions & 39 deletions

‎backend/docs/LOG_RETENTION.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Log Retention & Archival Service
2+
3+
`src/lib/log-retention.js` purges operational webhook delivery logs
4+
(`webhook_delivery_logs`) older than the retention window, in batches. Run it
5+
with `npm run purge:webhook-logs` (see `scripts/purge-webhook-logs.js` and
6+
`.github/workflows/webhook-log-retention.yml`).
7+
8+
> Compliance: confirm the table holds operational delivery telemetry only.
9+
> Payment audit records may need longer retention by policy or regulation.
10+
11+
## Configuration
12+
13+
| Option | Environment variable | Default | Allowed |
14+
|---|---|---|---|
15+
| `retentionDays` | `LOG_RETENTION_DAYS` | 30 | 1-3650 |
16+
| `batchSize` | `LOG_PURGE_BATCH_SIZE` | 1000 | 1-50000 |
17+
| `maxDurationMs` | `LOG_PURGE_MAX_DURATION_MS` | 300000 | 1000-86400000 |
18+
| `maxBatches` | `LOG_PURGE_MAX_BATCHES` | 100000 | 1-10000000 |
19+
20+
Explicit options win over environment variables. `logger` (an object with
21+
`log` and `error`) may be passed instead of `console`.
22+
23+
## Strict validation and sanitization (#1472)
24+
25+
- Numbers must be plain base-10 integers in range. Values such as `30days`,
26+
`1e3`, `-5`, `0x10` or `1.5` are rejected with a
27+
`LogRetentionValidationError` naming the option or environment variable.
28+
Before this change `parseInt` silently turned `30days` into `30`.
29+
- Unknown option keys (for example a typo like `retentiondays`) are rejected
30+
rather than silently falling back to the default retention.
31+
- `pool` must expose `query()`. Everything is validated before the first
32+
query runs.
33+
- Only allow-listed table/column pairs can be purged, and identifiers are
34+
checked against `^[a-z_][a-z0-9_]{0,62}$` before they reach SQL. The
35+
retention period and batch size are always bind parameters.
36+
- Each batch's `rowCount` must be an integer between 0 and `batchSize`; a
37+
bad value stops the purge. `maxBatches` caps the loop so a misbehaving
38+
driver cannot spin forever.
39+
- The returned summary is a frozen object with `tableName`, `retentionDays`,
40+
`batchSize`, `totalDeleted`, `durationMs` and `batches` only.
41+
- `sanitizeLogMessage()` masks credentials (`scheme://user:pass@`,
42+
`password=`, `token=`, `Bearer ...`), strips control characters (log
43+
injection) and caps length. The purge script logs only the scrubbed message
44+
and the error's name/code, never the raw error object, which can echo the
45+
database connection string.

‎backend/docs/PAYMENT_PROCESSOR.md‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,11 +110,40 @@ Include the printed percentile tables in PRs touching the processor.
110110

111111
---
112112

113-
## 5. Security notes
113+
## 5. Telemetry integration and stress suite (#1481)
114+
115+
| Suite | File | Run |
116+
|---|---|---|
117+
| Integration | `tests/integration/payment-processor-telemetry.test.js` | `npx vitest run tests/integration/payment-processor-telemetry.test.js` |
118+
| Stress | `load-tests/payment-processor-telemetry-stress.test.js` | `npm run test:load -- payment-processor-telemetry` |
119+
120+
The integration suite scrapes the real `/metrics` route and parses the
121+
exposition text a Prometheus server would receive. It checks that every
122+
`payment_processor_*` series is merged into the scrape, that counts are
123+
exact per asset and outcome, that histograms are consistent (cumulative
124+
buckets, `+Inf` equals `_count`, `_sum` matches), that concurrent scrapes
125+
during writes never fail, and that the output always parses.
126+
127+
The stress suite checks there are no lost increments (100 concurrent workers
128+
x 1,000 events), that cardinality stays bounded under 50,000 adversarial
129+
asset values, that histogram count and sum stay exact over 100,000
130+
observations, and that scrape p95 latency stays under 100 ms while writes
131+
continue.
132+
133+
**Fix found by the suite:** session creation used `body.asset` straight from
134+
the request as the `asset` label, so a client could create one time series
135+
per arbitrary string (including on `validation_failed`). Labels now go
136+
through `toAssetLabel()`: well-formed asset codes are upper-cased, anything
137+
else becomes `invalid`, and a missing value becomes `unknown`.
138+
139+
---
140+
141+
## 6. Security notes
114142

115143
- API keys are hashed for rate-limit bucketing (`getCreatePaymentRateLimitKey`)
116144
- Metadata keys must match `^[a-zA-Z0-9_-]{1,64}$`; unsafe keys never reach SQL
117145
- Refunds require a confirmed payment and are single-use (`refund_status`)
118146
- Verification uses a DB-level conditional update so a `tx_hash` can only be
119147
claimed once even under concurrent confirmation attempts
120148
- Signature verification failure keeps payments pending (fail-closed)
149+
- Metric labels never carry raw client input; `asset` is bounded by `toAssetLabel()`

‎backend/docs/WEBSOCKET_RELAY.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# WebSocket Relay Service
2+
3+
Covers issues **#1454** (automated retry with exponential backoff) and
4+
**#1453** (Prometheus alert metrics and health telemetry).
5+
6+
## 1. Layout
7+
8+
| File | Role |
9+
|---|---|
10+
| `src/lib/websocket-relay-security.js` | Origin, size, JWT and message-shape checks |
11+
| `src/lib/websocket-relay-recovery.js` | Circuit breaker, dead letter queue, `reconnectWithBackoff` |
12+
| `src/lib/websocket-relay-retry.js` | Retry policy, `deliverRelayMessage`, `redriveDeadLetters` (#1454) |
13+
| `src/lib/websocket-relay-metrics.js` | Granular relay counters (#1083) |
14+
| `src/lib/websocket-relay-telemetry.js` | Alert metrics and rolling-window health (#1453) |
15+
| `docs/alerts/websocket-relay.rules.yml` | Prometheus alert rules |
16+
17+
## 2. Retry with exponential backoff (#1454)
18+
19+
```js
20+
import { deliverRelayMessage, redriveDeadLetters, withRelayRetry } from "./lib/websocket-relay-retry.js";
21+
import { CircuitBreaker, DeadLetterQueue } from "./lib/websocket-relay-recovery.js";
22+
import { attachRelayHealthSource } from "./lib/websocket-relay-telemetry.js";
23+
24+
const circuitBreaker = new CircuitBreaker();
25+
const dlq = new DeadLetterQueue();
26+
attachRelayHealthSource({ circuitBreaker, dlq });
27+
28+
// Deliver, retrying transient failures, dead-lettering what still fails.
29+
await deliverRelayMessage(message, (msg) => socket.sendAsync(msg), { circuitBreaker, dlq });
30+
31+
// Reconnect with the same policy.
32+
await withRelayRetry(() => connect(), { operation: "reconnect", signal });
33+
34+
// Later, once the downstream recovers:
35+
await redriveDeadLetters(dlq, (msg) => socket.sendAsync(msg), { limit: 100 });
36+
```
37+
38+
- **Delay:** full jitter, `random(0, min(maxDelayMs, baseDelayMs * 2^n))`.
39+
- **Retried:** network error codes (`ECONNRESET`, `ETIMEDOUT`, ...), timeouts,
40+
HTTP 408/429/5xx (not 501), WebSocket close codes 1006/1011/1012/1013, and
41+
errors flagged `retryable: true`.
42+
- **Not retried:** an OPEN circuit breaker (`circuitOpen`), other 4xx
43+
(validation, signature, auth), policy close codes, `retryable: false`,
44+
and programming errors. These fail or dead-letter immediately.
45+
- **Bounds:** `maxAttempts`, a total `maxElapsedMs` budget including waits,
46+
and an optional `AbortSignal`.
47+
- **Failures** carry `retryAttempts`. With a `dlq`, `deliverRelayMessage`
48+
dead-letters the message (with the total retry count) instead of throwing.
49+
- **Logs** contain the operation label, attempt counts, delay and the
50+
error's code/status/close code, with the message truncated to 200
51+
characters. Payloads are never logged.
52+
53+
| Variable | Default | Range |
54+
|---|---|---|
55+
| `RELAY_RETRY_MAX_ATTEMPTS` | `5` | 1-10 |
56+
| `RELAY_RETRY_BASE_DELAY_MS` | `250` | 0-60000 |
57+
| `RELAY_RETRY_MAX_DELAY_MS` | `10000` | base-60000 |
58+
| `RELAY_RETRY_MAX_ELAPSED_MS` | `30000` | 0-300000 |
59+
60+
`reconnectWithBackoff` in `websocket-relay-recovery.js` is unchanged.
61+
62+
## 3. Alert metrics and health (#1453)
63+
64+
Exposed on `GET /metrics` (merged from their own registry, together with the
65+
existing `relay_*` series, which were not previously scraped).
66+
67+
| Metric | Type | Labels |
68+
|---|---|---|
69+
| `websocket_relay_deliveries_total` | counter | `outcome`: delivered, recovered, dead_lettered, failed |
70+
| `websocket_relay_retries_total` | counter | `operation`: deliver, reconnect, redrive, other; `result`: scheduled, recovered, exhausted, aborted |
71+
| `websocket_relay_retry_delay_seconds` | histogram | `operation` |
72+
| `websocket_relay_health_state` | gauge | 0 healthy, 1 degraded, 2 unhealthy |
73+
| `websocket_relay_delivery_failure_ratio` | gauge | rolling window |
74+
| `websocket_relay_dlq_depth` | gauge | live DLQ size |
75+
| `websocket_relay_circuit_open` | gauge | 1 OPEN, 0.5 HALF_OPEN, 0 CLOSED |
76+
| `websocket_relay_last_delivery_timestamp_seconds` | gauge | |
77+
78+
Gauges are computed at scrape time from a fixed ring of 5-second buckets, so
79+
they decay when traffic stops and memory stays constant.
80+
81+
`GET /health/websocket-relay` returns the snapshot (no payloads or payment
82+
ids). Status rules:
83+
84+
| Status | When |
85+
|---|---|
86+
| unhealthy (HTTP 503) | circuit OPEN, failure ratio >= `RELAY_FAILURE_RATIO_THRESHOLD` (default 0.1) with at least `RELAY_HEALTH_MIN_SAMPLES` (20) deliveries, or DLQ >= `RELAY_DLQ_UNHEALTHY_THRESHOLD` (100) |
87+
| degraded (HTTP 200) | circuit HALF_OPEN, DLQ >= `RELAY_DLQ_DEGRADED_THRESHOLD` (1), or retries exhausted in the window |
88+
| healthy | otherwise |
89+
90+
The window is `RELAY_HEALTH_WINDOW_MS` (default 5 minutes). `GET /health`
91+
also reports `services.websocket_relay` (informational; it does not change
92+
the overall status code).
93+
94+
## 4. Security notes
95+
96+
- Label values come from fixed server-side sets; message fields, origins,
97+
payment ids and error text never become labels.
98+
- The health snapshot and logs contain counts, states and error codes only.
99+
- An OPEN breaker is never retried, so retries cannot defeat it.
100+
- Retry configuration is clamped, so a misconfigured environment cannot
101+
disable backoff or make a delivery wait for minutes.

‎backend/docs/alerts/websocket-relay.rules.yml‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Prometheus alerting rules for the WebSocket Relay Service (issue #1453).
2+
#
3+
# Load with: rule_files: ["docs/alerts/websocket-relay.rules.yml"]
4+
# Validate: promtool check rules docs/alerts/websocket-relay.rules.yml
5+
#
6+
# Metric reference: docs/WEBSOCKET_RELAY.md
7+
8+
groups:
9+
- name: websocket-relay
10+
rules:
11+
- alert: WebSocketRelayUnhealthy
12+
expr: max(websocket_relay_health_state) >= 2
13+
for: 5m
14+
labels:
15+
severity: critical
16+
annotations:
17+
summary: WebSocket relay reports unhealthy
18+
description: >
19+
The circuit breaker is open, the delivery failure ratio is above
20+
threshold, or the dead letter queue has a large backlog. See
21+
GET /health/websocket-relay for the reasons.
22+
23+
- alert: WebSocketRelayCircuitOpen
24+
expr: max(websocket_relay_circuit_open) >= 1
25+
for: 2m
26+
labels:
27+
severity: critical
28+
annotations:
29+
summary: WebSocket relay circuit breaker is open
30+
description: >
31+
Deliveries are being rejected without being attempted and go
32+
straight to the dead letter queue.
33+
34+
- alert: WebSocketRelayHighFailureRatio
35+
expr: |
36+
sum(rate(websocket_relay_deliveries_total{outcome=~"failed|dead_lettered"}[10m]))
37+
/
38+
clamp_min(sum(rate(websocket_relay_deliveries_total[10m])), 1e-9)
39+
> 0.1
40+
and sum(increase(websocket_relay_deliveries_total[10m])) >= 20
41+
for: 10m
42+
labels:
43+
severity: critical
44+
annotations:
45+
summary: More than 10% of WebSocket relay deliveries are failing
46+
description: >
47+
Check logs for "WebSocket relay operation failed after exhausting
48+
retries".
49+
50+
- alert: WebSocketRelayRetriesExhausted
51+
expr: sum(increase(websocket_relay_retries_total{result="exhausted"}[10m])) > 5
52+
for: 5m
53+
labels:
54+
severity: warning
55+
annotations:
56+
summary: WebSocket relay retries are being exhausted
57+
description: >
58+
Transient failures are lasting longer than the retry budget
59+
(RELAY_RETRY_MAX_ATTEMPTS / RELAY_RETRY_MAX_ELAPSED_MS).
60+
61+
- alert: WebSocketRelayDeadLetterBacklog
62+
expr: max(websocket_relay_dlq_depth) > 0
63+
for: 15m
64+
labels:
65+
severity: warning
66+
annotations:
67+
summary: WebSocket relay dead letter queue is not draining
68+
description: >
69+
Messages have been waiting in the DLQ for 15 minutes. Run
70+
redriveDeadLetters() once the downstream recovers.
71+
72+
- alert: WebSocketRelayNoDeliveries
73+
expr: |
74+
time() - max(websocket_relay_last_delivery_timestamp_seconds) > 1800
75+
and sum(increase(websocket_relay_deliveries_total[30m])) > 0
76+
for: 5m
77+
labels:
78+
severity: warning
79+
annotations:
80+
summary: WebSocket relay has attempted deliveries but none succeeded for 30 minutes
81+
description: >
82+
Deliveries are being attempted but none succeed. Check the
83+
relay_errors_total breakdown and the circuit breaker state.

0 commit comments

Comments
 (0)