Merge pull request #1603 from Y33t-dev/feature/be-sep0001-hardening-relay-test-suite-idempotency-retry

feat(backend): harden and instrument SEP-0001 generator, add relay integration/stress suite, retry idempotency Redis calls with backoff

Author: emdevelopa

backend/SEP0001_GENERATOR_GUIDE.md

@@ -36,7 +36,7 @@ curl https://api.example.com/.well-known/stellar.toml?merchant_id=550e8400-e29b-
 NETWORK_PASSPHRASE = "Test SDF Network ; September 2015"
 TRANSFER_SERVER = "https://api.example.com/api"
 DOCUMENTATION = "https://api.example.com/api-docs"
-ACCOUNTS = ["GBUQWP3BOUZX34ULNQG23RQ6F4YUSXHTQSXUSMIQSTBE2BRUY4DQAT2B"]
+ACCOUNTS = ["GDVEU3DD4KOFECV66VIHWEZOYX4ZKR3WV27L464SIIPOU2IUI3JCZA57"]
 
 [ORG]
 name = "Acme Payment Services"
@@ -56,7 +56,7 @@ The stellar.toml is generated from:
    - `API_BASE_URL`: Base URL of the API (default: "http://localhost:4000")
    - `FEDERATION_SERVER_URL`: Optional federation server
    - `DOCS_URL`: Documentation URL (default: `{API_BASE_URL}/api-docs`)
-   - `SIGNING_KEY`: Optional signing key for transactions
+   - `SIGNING_KEY`: Optional **public** signing key (`G...`). A secret seed (`S...`) is never published.
 
 2. **Merchant Settings** (from database):
    - `business_name`: Organization name
@@ -79,7 +79,7 @@ API_BASE_URL="https://api.example.com"
 TRANSFER_SERVER_URL="https://api.example.com/api"
 FEDERATION_SERVER_URL="https://federation.example.com"
 DOCS_URL="https://docs.example.com"
-SIGNING_KEY="GBUQWP3BOUZX34ULNQG23RQ6F4YUSXHTQSXUSMIQSTBE2BRUY4DQAT2B"
+SIGNING_KEY="GDVEU3DD4KOFECV66VIHWEZOYX4ZKR3WV27L464SIIPOU2IUI3JCZA57"
 ```
 
 ## Merchant Configuration
@@ -92,7 +92,7 @@ Merchants can configure their stellar.toml information through:
    - `notification_email`: Support email
 
 2. **Stellar Account** (set during registration):
-   - `recipient`: Your Stellar account address (e.g., `GBUQWP3BOUZX34ULNQG23RQ6F4YUSXHTQSXUSMIQSTBE2BRUY4DQAT2B`)
+   - `recipient`: Your Stellar account address (e.g., `GDVEU3DD4KOFECV66VIHWEZOYX4ZKR3WV27L464SIIPOU2IUI3JCZA57`)
 
 3. **Branding Configuration** (via merchant settings):
    - `branding_config.homepage`: Your website URL
@@ -129,13 +129,13 @@ curl -X POST https://api.example.com/api/merchant-branding \
   - Example: `"https://federation.example.com"`
 
 - **ACCOUNTS**: List of Stellar accounts operated by the merchant
-  - Example: `["GBUQWP3BOUZX34ULNQG23RQ6F4YUSXHTQSXUSMIQSTBE2BRUY4DQAT2B"]`
+  - Example: `["GDVEU3DD4KOFECV66VIHWEZOYX4ZKR3WV27L464SIIPOU2IUI3JCZA57"]`
 
 - **DOCUMENTATION**: URL to API documentation
   - Example: `"https://api.example.com/api-docs"`
 
 - **SIGNING_KEY**: Public key for transaction signing
-  - Example: `"GBUQWP3BOUZX34ULNQG23RQ6F4YUSXHTQSXUSMIQSTBE2BRUY4DQAT2B"`
+  - Example: `"GDVEU3DD4KOFECV66VIHWEZOYX4ZKR3WV27L464SIIPOU2IUI3JCZA57"`
 
 ### Organization Section
 
@@ -151,10 +151,64 @@ The `[ORG]` section contains merchant information:
 
 The stellar.toml is validated to ensure:
 
-1. **Required fields present**: NETWORK_PASSPHRASE and TRANSFER_SERVER
+1. **Required fields present**: NETWORK_PASSPHRASE and TRANSFER_SERVER, each at the start of a line
 2. **Valid TOML syntax**: Proper formatting and escaping
 3. **Valid URLs**: Properly formatted URLs
 4. **Valid Stellar addresses**: Properly formatted account addresses
+5. **Size**: the document is at most 100 KB (the SEP-0001 limit)
+
+### Payload sanitization and strict validation (issue #1457)
+
+Every value comes from the merchant record or server environment, and the
+generator treats all of it as untrusted.
+
+| Input | Rule | If invalid |
+|---|---|---|
+| `merchant_id` query param | Single value, must be a UUID | `400`. The database is not queried. |
+| `business_name` | Bidi override characters (U+202A–202E, U+2066–2069, U+200E/F) stripped, trimmed, max 256 characters | Falls back to `Stellar Payment Merchant` when empty or not a string |
+| All string values | TOML-escaped, including `\r`, `\t`, and every other control character (`\uXXXX`) | A value can never close its quotes and inject keys or `[TABLES]` |
+| `recipient` → `ACCOUNTS` | Valid Stellar public key, StrKey checksum included | `ACCOUNTS` omitted |
+| `SIGNING_KEY` | Valid Stellar **public** key | Omitted. A secret seed is never published and is logged as an error (without the value). |
+| `email`, `notification_email` | Email syntax, max 254 characters | Field omitted |
+| `branding_config.homepage` / `logo_url` | `https` URL, no credentials, no whitespace, max 2048 characters | Field omitted |
+| `TRANSFER_SERVER_URL` / `API_BASE_URL` | `https` URL. `http` is allowed only for `localhost`/`127.0.0.1`. No credentials. | `500 Failed to generate valid stellar.toml`. Details are logged, never returned. |
+| `FEDERATION_SERVER_URL`, `DOCS_URL` | Same URL rules | Field omitted |
+| `STELLAR_NETWORK_PASSPHRASE` | Printable, max 128 characters | `500`, as above |
+
+Omitted or altered fields are logged (`SEP-0001: fields omitted or altered by
+sanitization`) and counted in `sep0001_rejected_fields_total{field,reason}`.
+
+Responses also send `Access-Control-Allow-Origin: *`, which SEP-0001 requires
+so browser wallets can read the file, and `X-Content-Type-Options: nosniff`.
+
+## Monitoring (issue #1458)
+
+Metrics (merged into `GET /metrics`):
+
+| Metric | Labels | Meaning |
+|---|---|---|
+| `sep0001_requests_total` | `outcome` = `success` / `invalid_request` / `not_found` / `lookup_error` / `generation_error` | Requests by outcome |
+| `sep0001_generation_duration_seconds` | `outcome` | Lookup + generation latency |
+| `sep0001_document_size_bytes` | — | Served document size |
+| `sep0001_rejected_fields_total` | `field`, `reason` | Sanitization rejections |
+| `sep0001_health_state` | — | `0` healthy, `1` degraded, `2` unhealthy |
+| `sep0001_error_ratio` | — | Rolling-window server-error ratio |
+| `sep0001_last_success_timestamp_seconds` | — | Last successful response |
+
+All label values come from fixed sets. Merchant ids never become labels.
+
+Health: `GET /health/sep0001` returns the rolling-window snapshot (5 minutes by
+default). It responds `503` only when unhealthy. `/health` also reports
+`services.sep0001_generator` for information.
+
+| Status | Condition (with at least `SEP0001_HEALTH_MIN_SAMPLES`, default 10) |
+|---|---|
+| `unhealthy` | server-error ratio (`lookup_error` + `generation_error`) ≥ `SEP0001_UNHEALTHY_ERROR_RATIO` (default 0.25) |
+| `degraded` | server-error ratio ≥ `SEP0001_DEGRADED_ERROR_RATIO` (default 0.05), or a secret seed was refused as `SIGNING_KEY` in the window |
+
+`400` and `404` responses are client errors and do not affect health. The
+window is set by `SEP0001_HEALTH_WINDOW_MS`. Alerting rules are in
+`docs/alerts/sep0001-generator.rules.yml`.
 
 ## Caching
 
@@ -267,6 +321,9 @@ console.log('Accounts:', toml.ACCOUNTS);
 1. **Public Information**: stellar.toml is publicly accessible
    - Don't include sensitive information
    - Only include information you want to share publicly
+   - `SIGNING_KEY` must be the public key. If a secret seed is configured, the
+     generator refuses to publish it and `Sep0001SecretSigningKeyRefused` fires.
+     Treat that seed as exposed in config and rotate it.
 
 2. **HTTPS Required**: Always use HTTPS for stellar.toml
    - Wallets will reject HTTP endpoints

backend/docs/IDEMPOTENCY_KEY_MANAGER.md

@@ -0,0 +1,60 @@
+# Idempotency Key Manager
+
+`src/lib/idempotency.js` makes `POST` requests with an `Idempotency-Key`
+header safe to retry. The first 2xx response is cached in Redis for 24 hours
+under `idempotency:<merchant>:<key>` together with a SHA-256 hash of the
+request body. A later request with the same key and the same body replays the
+cached response (`201`). The same key with a different body gets a `400`.
+
+## Retry with exponential backoff (issue #1464)
+
+Every Redis call the manager makes goes through `withIdempotencyRetry()`
+(`src/lib/idempotency-retry.js`):
+
+| Operation label | Call | On final failure |
+|---|---|---|
+| `get` | middleware cache read | Request proceeds without idempotency (fail-open, unchanged) |
+| `set` | caching the response | Logged. Runs in the background and never delays the response. |
+| `lookup` | `lookupIdempotentResponse()` (payment session handler) | Error propagates to the caller, unchanged |
+
+```
+delay(attempt) = min(baseDelayMs * 2^attempt, maxDelayMs) * (1 ± jitterRatio)
+```
+
+Retries stop at `maxRetries`, or when the next delay would push total elapsed
+time past `totalBudgetMs`, so a Redis outage adds bounded latency.
+
+Only transient errors are retried: socket closed or reset, connect timeouts,
+`ECONNRESET`/`ETIMEDOUT`/..., and the server replies `LOADING`, `BUSY`,
+`TRYAGAIN`, `READONLY`, `MASTERDOWN` and `CLUSTERDOWN`. A closed client
+(`redis.js` disables reconnects), auth/permission/`WRONGTYPE` errors and
+corrupt cache JSON fail immediately.
+
+| Env var | Default | Bounds |
+|---|---|---|
+| `IDEMPOTENCY_REDIS_MAX_RETRIES` | 2 | 0–5 |
+| `IDEMPOTENCY_REDIS_RETRY_BASE_MS` | 25 | 1–1000 |
+| `IDEMPOTENCY_REDIS_RETRY_MAX_MS` | 250 | base–5000 |
+| `IDEMPOTENCY_REDIS_RETRY_BUDGET_MS` | 500 | 0–2000 |
+
+Jitter is fixed at ±20%.
+
+## Metrics
+
+Exposed on `GET /metrics`, from their own registry:
+
+| Metric | Labels |
+|---|---|
+| `idempotency_redis_operations_total` | `operation` (`get`/`set`/`lookup`), `outcome` (`success`/`recovered`/`failed`/`non_retryable`) |
+| `idempotency_redis_retries_total` | `operation` |
+| `idempotency_redis_retry_delay_seconds` | `operation` |
+
+A rising `outcome="failed"` rate for `get` means idempotency is currently
+failing open, so duplicate payments are possible. Alert on it together with
+Redis availability.
+
+## Tests
+
+```bash
+npx vitest run src/lib/idempotency-retry.test.js src/lib/idempotency.test.js
+```

backend/docs/WEBSOCKET_RELAY_TESTING.md

@@ -0,0 +1,29 @@
+# WebSocket Relay Service: integration and stress tests
+
+Issue #1456.
+
+| Suite | File | What it proves |
+|---|---|---|
+| Integration | `src/lib/websocket-relay.integration.test.js` | The real security, query, recovery and metrics modules wired as a relay worker: handshake (origin allow-list, HS256-pinned JWT, `alg: none`/expired/wrong-secret rejection), frame size limit, unknown/prototype-polluting field stripping, parameterised batch insert, FIFO dequeue, delivery through the circuit breaker, `markEventComplete`/`markEventFailed`, dead-lettering, HALF_OPEN recovery and DLQ replay, DB failure wrapping, pagination clamps, reconnect backoff, and audit log integrity |
+| Stress | `src/lib/websocket-relay.stress.test.js` | 50 concurrent consumers drain 2,000 events with every event claimed exactly once; interleaved async latency never double-claims; a 1,000-row single-statement insert; 10,000 frames sanitized; oversized-frame floods; byte-accurate size checks for multi-byte text; 1,000 concurrent fast-fails on an open circuit; 10,000 mixed breaker outcomes; a 10,000-entry DLQ in FIFO order; 200 concurrent reconnect loops staying under the delay cap |
+
+Only Postgres is faked. `tests/helpers/fake-relay-db.js` interprets the exact
+SQL emitted by `websocket-relay-queries.js`. Each statement applies
+atomically, mirroring `FOR UPDATE SKIP LOCKED` row locking, so the
+exactly-once assertions test the claiming logic rather than the fake.
+
+Timing assertions in the stress suite are deliberately loose. They catch
+pathological regressions (quadratic or blocking behaviour), not small slowdowns.
+
+## Finding fixed
+
+`auditRelayEvent()` spread caller metadata **after** the fixed fields, so
+metadata containing `audit`, `ts` or `event` (for example values echoed from
+a client) could overwrite them and forge an audit line. Metadata is now
+spread first.
+
+## Running
+
+```bash
+npx vitest run src/lib/websocket-relay
+```

backend/docs/alerts/sep0001-generator.rules.yml

@@ -0,0 +1,63 @@
+# Prometheus alerting rules for the SEP-0001 stellar.toml generator (issue #1458).
+#
+# Load with:   rule_files: ["docs/alerts/sep0001-generator.rules.yml"]
+# Validate:    promtool check rules docs/alerts/sep0001-generator.rules.yml
+#
+# Metric reference: SEP0001_GENERATOR_GUIDE.md ("Monitoring")
+
+groups:
+  - name: sep0001-generator
+    rules:
+      - alert: Sep0001GeneratorUnhealthy
+        expr: max(sep0001_health_state) >= 2
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: stellar.toml generation is failing
+          description: >
+            The rolling-window server error ratio is above the unhealthy
+            threshold. Wallets cannot discover TRANSFER_SERVER or verify
+            merchant accounts. See GET /health/sep0001 and logs for
+            "SEP-0001: failed to fetch merchant" / "invalid server configuration".
+
+      - alert: Sep0001GeneratorDegraded
+        expr: max(sep0001_health_state) == 1
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          summary: stellar.toml generator degraded for 15 minutes
+          description: Elevated error ratio or a refused secret SIGNING_KEY. See GET /health/sep0001.
+
+      - alert: Sep0001SecretSigningKeyRefused
+        expr: sum(increase(sep0001_rejected_fields_total{field="SIGNING_KEY",reason="secret_seed"}[15m])) > 0
+        for: 0m
+        labels:
+          severity: critical
+        annotations:
+          summary: SIGNING_KEY is configured with a secret seed
+          description: >
+            The generator refused to publish it, but the seed is present in the
+            environment. Rotate the key and set SIGNING_KEY to the public key.
+
+      - alert: Sep0001InvalidConfiguration
+        expr: sum(increase(sep0001_requests_total{outcome="generation_error"}[10m])) > 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: stellar.toml cannot be generated
+          description: >
+            Usually TRANSFER_SERVER_URL / API_BASE_URL is not https or
+            STELLAR_NETWORK_PASSPHRASE is invalid. Check logs for
+            "SEP-0001: invalid server configuration".
+
+      - alert: Sep0001MerchantDataRejected
+        expr: sum by (field, reason) (increase(sep0001_rejected_fields_total{field!="SIGNING_KEY"}[1h])) > 20
+        for: 0m
+        labels:
+          severity: info
+        annotations:
+          summary: "Many stellar.toml {{ $labels.field }} values rejected ({{ $labels.reason }})"
+          description: Merchant records contain values that are being omitted from stellar.toml.