Merge pull request #1610 from Abidoyesimze/feature/be-add-prometheus-alert-metrics-and-health-telemetry-to-database-row-level-security-service

feat(backend): add Prometheus alert metrics and health telemetry to Database Row-Level Security Service

Author: emdevelopa

backend/docs/DATABASE_ROW_LEVEL_SECURITY.md

@@ -0,0 +1,93 @@
+# Database Row-Level Security Service — Telemetry
+
+Issue #1468. Every merchant-scoped transaction runs through `withMerchantContext()`
+(`src/lib/db-rls.js`), which sets `app.current_merchant_id` inside a transaction so
+Postgres Row-Level Security policies apply. This document covers the Prometheus
+metrics, health endpoint and alert rules added for that service.
+
+## Metrics
+
+Exported from `src/lib/db-rls-telemetry.js` on its own registry, merged into the
+main `/metrics` scrape by `src/routes/prometheus.js`. Label values are a fixed
+server-side set — never merchant ids, SQL text or error strings — so request
+traffic cannot inflate series cardinality or leak customer data into scrapes.
+
+| Metric | Type | Labels | Meaning |
+| --- | --- | --- | --- |
+| `database_rls_transactions_total` | Counter | `outcome` | Transaction attempts by outcome |
+| `database_rls_transaction_duration_seconds` | Histogram | `outcome` | Wall time of the whole `withMerchantContext` call |
+| `database_rls_active_transactions` | Gauge | — | Transactions currently in flight |
+| `database_rls_health_state` | Gauge | — | `0` healthy, `1` degraded, `2` unhealthy (rolling window) |
+| `database_rls_error_ratio` | Gauge | — | Share of attempts that failed before completing |
+| `database_rls_rollback_ratio` | Gauge | — | Share of attempts rolled back by a failing callback |
+| `database_rls_last_transaction_timestamp_seconds` | Gauge | — | Unix time of the most recent attempt |
+
+`outcome` values:
+
+- `committed` — transaction committed successfully
+- `rolled_back` — the caller's callback threw; the transaction was rolled back
+- `invalid_input` — merchantId or callback failed validation; the pool was never touched
+- `connection_error` — `pool.connect()` failed (pool exhausted, DB down)
+- `transaction_error` — `BEGIN` / `SET LOCAL` / `COMMIT` failed, or `ROLLBACK` itself failed
+
+## Health endpoint
+
+`GET /health/database-row-level-security` returns the rolling-window snapshot:
+
+```json
+{
+  "status": "healthy",
+  "reasons": [],
+  "window_ms": 300000,
+  "total": 1421,
+  "committed": 1418,
+  "rolled_back": 3,
+  "invalid_input": 0,
+  "connection_error": 0,
+  "transaction_error": 0,
+  "errors": 0,
+  "rollback_ratio": 0.0021,
+  "error_ratio": 0,
+  "last_transaction_at": "2026-10-17T12:00:00.000Z",
+  "thresholds": {
+    "min_samples": 20,
+    "error_ratio": 0.05,
+    "rollback_ratio": 0.5,
+    "invalid_input": 10
+  }
+}
+```
+
+Returns 503 only when `status` is `unhealthy`; `degraded` still returns 200. The
+same status is reported (informationally) under `services.database_row_level_security`
+in the main `GET /health` payload. The body contains no merchant ids, query text
+or error strings.
+
+Status rules over the rolling window (default 5 minutes, 20-sample minimum):
+
+- **unhealthy** — connection + transaction errors ≥ 5% of attempts
+- **degraded** — rolled-back callbacks ≥ 50% of attempts, or ≥ 10 `invalid_input` outcomes
+- **healthy** — everything else
+
+Thresholds are overridable with `DATABASE_RLS_HEALTH_WINDOW_MS`,
+`DATABASE_RLS_HEALTH_MIN_SAMPLES`, `DATABASE_RLS_ERROR_RATIO_THRESHOLD`,
+`DATABASE_RLS_ROLLBACK_RATIO_THRESHOLD` and `DATABASE_RLS_INVALID_INPUT_THRESHOLD`.
+
+## Alert rules
+
+`docs/alerts/database-row-level-security.rules.yml` — load with
+`rule_files` and validate with
+`promtool check rules docs/alerts/database-row-level-security.rules.yml`:
+
+- `DatabaseRlsUnhealthy` (critical) — health state ≥ 2 for 5m
+- `DatabaseRlsHighErrorRatio` (critical) — > 5% connection/transaction errors over 10m
+- `DatabaseRlsHighRollbackRatio` (warning) — > 50% callback rollbacks over 15m
+- `DatabaseRlsInvalidInputSpike` (warning) — > 10 invalid merchant ids in 10m (a caller is misbehaving)
+- `DatabaseRlsNoTraffic` (info) — no RLS transactions for 2h
+
+## Notes on privacy
+
+Telemetry records only aggregate counts, durations and derived ratios. Merchant
+ids are never used as label values or logged into the health snapshot, so the
+scrape and health outputs stay safe to expose to the same monitoring stack as the
+rest of the API.

backend/docs/alerts/database-row-level-security.rules.yml

@@ -0,0 +1,81 @@
+# Prometheus alerting rules for the Database Row-Level Security Service (issue #1468).
+#
+# Load with:   rule_files: ["docs/alerts/database-row-level-security.rules.yml"]
+# Validate:    promtool check rules docs/alerts/database-row-level-security.rules.yml
+#
+# Metric reference: docs/DATABASE_ROW_LEVEL_SECURITY.md
+
+groups:
+  - name: database-row-level-security
+    rules:
+      - alert: DatabaseRlsUnhealthy
+        expr: max(database_rls_health_state) >= 2
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: Database Row-Level Security service reports unhealthy
+          description: >
+            Rolling-window internal error ratio is above threshold —
+            connections or transactions are failing before commit. See
+            GET /health/database-row-level-security for counts and reasons.
+
+      - alert: DatabaseRlsHighErrorRatio
+        expr: |
+          (
+              sum(rate(database_rls_transactions_total{outcome="connection_error"}[10m]))
+            + sum(rate(database_rls_transactions_total{outcome="transaction_error"}[10m]))
+          )
+            /
+          clamp_min(sum(rate(database_rls_transactions_total[10m])), 1e-9)
+            > 0.05
+          and sum(increase(database_rls_transactions_total[10m])) >= 20
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          summary: More than 5% of RLS transactions fail before completing
+          description: >
+            Check logs for "Failed to acquire pooled connection" or
+            "Failed to rollback transaction", and inspect Postgres
+            connection limits and transaction timeouts.
+
+      - alert: DatabaseRlsHighRollbackRatio
+        expr: |
+          sum(rate(database_rls_transactions_total{outcome="rolled_back"}[15m]))
+            /
+          clamp_min(sum(rate(database_rls_transactions_total[15m])), 1e-9)
+            > 0.5
+          and sum(increase(database_rls_transactions_total[15m])) >= 20
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          summary: Majority of RLS transactions are rolled back by failing callbacks
+          description: >
+            withMerchantContext is rolling back most transactions, which usually
+            means merchant-scoped queries are erroring (bad SQL, constraint
+            violations, or validation inside the callback).
+
+      - alert: DatabaseRlsInvalidInputSpike
+        expr: sum(increase(database_rls_transactions_total{outcome="invalid_input"}[10m])) > 10
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: Spike of invalid merchant ids reaching the RLS service
+          description: >
+            Callers are passing merchant ids that fail sanitization/validation
+            (empty, too long, or illegal characters). Find the offending caller
+            in the logs ("Invalid merchantId provided to withMerchantContext").
+
+      - alert: DatabaseRlsNoTraffic
+        expr: sum(increase(database_rls_transactions_total[1h])) == 0
+        for: 2h
+        labels:
+          severity: info
+        annotations:
+          summary: No RLS transactions observed for two hours
+          description: >
+            The service may be unused in this environment, or the RLS path
+            has stopped being exercised. Confirm with deployment state.

backend/src/app.js

@@ -49,6 +49,7 @@ import { versionDeprecationMiddleware } from "./lib/version-deprecation.js";
 import oracleRouter from "./routes/oracle.js";
 import { getPaymentSessionValidatorHealth } from "./lib/payment-session-validator.js";
 import { getExchangeRateOracleHealth } from "./lib/exchange-rate-oracle-telemetry.js";
+import { getDatabaseRowLevelSecurityHealth } from "./lib/db-rls-telemetry.js";
 import { configureExchangeRateCoordination } from "./services/exchangeRateService.js";
 
 export async function createApp({ redisClient }) {
@@ -262,6 +263,7 @@ export async function createApp({ redisClient }) {
         // Informational only — does not affect `ok` / the status code.
         payment_session_validator: getPaymentSessionValidatorHealth().status,
         exchange_rate_oracle_cache: getExchangeRateOracleHealth().status,
+        database_row_level_security: getDatabaseRowLevelSecurityHealth().status,
       },
     });
   });
@@ -314,6 +316,31 @@ export async function createApp({ redisClient }) {
     res.status(health.status === "unhealthy" ? 503 : 200).json(health);
   });
 
+  /**
+   * @swagger
+   * /health/database-row-level-security:
+   *   get:
+   *     summary: Database Row-Level Security Service health telemetry
+   *     description: >
+   *       Rolling-window counts, error / rollback ratios and derived status
+   *       for transactions executed through withMerchantContext (issue #1468).
+   *       Returns 503 only when the service is unhealthy (internal error ratio
+   *       above threshold); a degraded status (high rollback ratio or an
+   *       invalid-input spike) still returns 200. The body contains no
+   *       merchant ids, query text or error strings.
+   *     tags: [Health]
+   *     security: []
+   *     responses:
+   *       200:
+   *         description: RLS service healthy or degraded
+   *       503:
+   *         description: RLS service unhealthy
+   */
+  app.get("/health/database-row-level-security", (_req, res) => {
+    const health = getDatabaseRowLevelSecurityHealth();
+    res.status(health.status === "unhealthy" ? 503 : 200).json(health);
+  });
+
   const verifyPaymentRateLimit = createVerifyPaymentRateLimit({
     store: redisAvailable ? createRedisRateLimitStore({ client: redisClient }) : undefined,
   });