diff --git a/.github/workflows/converged-modeling-patterns.yml b/.github/workflows/converged-modeling-patterns.yml new file mode 100644 index 0000000..7a8bfac --- /dev/null +++ b/.github/workflows/converged-modeling-patterns.yml @@ -0,0 +1,41 @@ +name: converged-modeling-patterns +on: + push: + branches: [main] + paths: + - 'labs/converged-modeling-patterns/**' + - '.github/workflows/converged-modeling-patterns.yml' + pull_request: + paths: + - 'labs/converged-modeling-patterns/**' + - '.github/workflows/converged-modeling-patterns.yml' + schedule: + - cron: '41 6 * * *' # nightly: substantive freshness, not cosmetic + workflow_dispatch: + +jobs: + validate: + runs-on: ubuntu-latest + # Generous: this lab builds a large image (Oracle AI Database 26ai Free + ORDS + # + mongosh) before validating both lanes. + timeout-minutes: 55 + defaults: + run: + working-directory: labs/converged-modeling-patterns + steps: + - uses: actions/checkout@v4 + + - name: Free disk space + run: sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL + + - name: Build the lab images (DB + ORDS + mongosh, and the hands-on console) + run: docker compose build + + - name: Validate both lanes (SQL + Oracle API for MongoDB) + # run.sh brings the stack up, waits for the database AND the MongoDB API, + # then runs every pattern across BOTH lanes — grading each .sql and .js; + # the cross-API parity scripts exit non-zero on mismatch. It ends with a + # combined "N passed, 0 failed" and a non-zero exit if anything failed. + # Stage 3 then runs the hands-on console's unit + integration suite in a + # container against the same database. + run: ./run.sh diff --git a/labs/README.md b/labs/README.md index 9d6c56f..339626c 100644 --- a/labs/README.md +++ b/labs/README.md @@ -6,5 +6,6 @@ Self-contained, runnable labs that validate converged-model claims against a liv | --- | --- | --- | | converged-database-lab | Runnable proofs for the converged-database article series — JSON Relational Duality, single-table vs. converged modeling, graph, vector, spatial, and full-text claims, each executing against a free Oracle AI Database 26ai container. | [./converged-database-lab](./converged-database-lab) | | oracle-database-kafka-apis | Java integration tests demonstrating Oracle AI Database Transactional Event Queues through the Kafka APIs, backed by Testcontainers. | [./oracle-database-kafka-apis](./oracle-database-kafka-apis) | +| converged-modeling-patterns | Six document-modeling patterns as runnable side-by-sides — the document-model starting point and the converged alternative that keeps the read win without the write amplification — validated across SQL and the Oracle API for MongoDB with cross-API parity assertions, on a free Oracle AI Database 26ai container, plus a browser-based hands-on console (run and edit every query, measure document-model vs converged write cost live). | [./converged-modeling-patterns](./converged-modeling-patterns) | *More labs coming as the content series expands.* diff --git a/labs/converged-modeling-patterns/.dockerignore b/labs/converged-modeling-patterns/.dockerignore new file mode 100644 index 0000000..2508d0b --- /dev/null +++ b/labs/converged-modeling-patterns/.dockerignore @@ -0,0 +1,9 @@ +# .dockerignore (repo root) +.git +.superpowers/ +.env +app/node_modules +app/public/vendor +app/test/artifacts +docs +node_modules diff --git a/labs/converged-modeling-patterns/.gitignore b/labs/converged-modeling-patterns/.gitignore new file mode 100644 index 0000000..6350ade --- /dev/null +++ b/labs/converged-modeling-patterns/.gitignore @@ -0,0 +1,10 @@ +# OS / editor +.DS_Store +*.swp +*~ + +# local env overrides +.env + +# nothing generated is checked in; the DB lives in a docker volume +.superpowers/ diff --git a/labs/converged-modeling-patterns/README.md b/labs/converged-modeling-patterns/README.md new file mode 100644 index 0000000..fe35dfc --- /dev/null +++ b/labs/converged-modeling-patterns/README.md @@ -0,0 +1,415 @@ +# Converged Modeling Patterns + +Companion repository for the 90-minute lecture **"Model the Domain, Not the Engine: +Converged Data Modeling."** Every document-modeling pattern in this repo is a +runnable, side-by-side exploration of a tradeoff: the document-model design a developer +would build, a converged alternative, and what each one costs on reads **and** on +writes. Sometimes the converged alternative wins; sometimes the document shape should +stay (see *Where document still wins*). The lab measures where the needle flips +instead of assuming it. + +Runs on **Oracle AI Database 26ai Free**: one container, one command. Two lanes: +the same data through **SQL** (`sqlplus`) and through the **Oracle API for MongoDB** +(`mongosh`), with automated assertions that the two lanes return **byte-identical** +results. One truth, many shapes. Proven, not asserted. + +> **Data modeling is a physics problem, not a philosophy problem.** The famous +> document patterns are not folklore. They are disciplined responses to real +> storage-engine constraints. This repo asks one question of each: *does that +> constraint still exist on a converged engine?* When it does, keep the pattern. +> When it was a workaround for a missing capability, there is a better shape. + +--- + +## The one idea + +Every pattern trades along the same axis: **what you access together, you store +together.** Embed and you get atomic reads, until the document grows, hits 16 MB, or +takes write amplification on every change. Reference and you get small writes, but +now you need a join the engine may not love. + +Convergence adds a **third resolution the catalog never had: _project it._** Model +the domain once, as its **canonical form**: the entities, properties and relationships +that are the primary truth. Store that truth as one or more **projections**, chosen per +access pattern: rows on tables, embedded documents, or graph nodes and edges. Each +consumer's query, processed by an **access surface** (SQL, SQL/JSON, SQL/PGQ or the +MongoDB API), creates the **projected shape** it needs (document, graph, time-series, +relational) at read time, in the same transaction. Relational is one projection among +several, not the center: **the workload picks the shape**. + +The vocabulary used throughout (Unified Model Theory, or UMT): + +| Term | Meaning | +|---|---| +| **Canonical form** | the primary truth: the logical model of entities, properties and relationships, each fact stated once. Not "the tables" | +| **Projection** | the stored form of that truth: rows on tables, embedded documents (JSON collections, duality views), graph nodes and edges | +| **Access surface** | the interface a query is processed by: SQL, SQL/JSON, SQL/PGQ, the MongoDB API | +| **Projected shape** | what a query creates when an access surface processes it: a document, a graph, a rowset, a time series | +| **Access dimension** | the kind of question the query asks: point read, range, aggregate, traverse | + +It is the query, not the storage, that makes the shape. + +No projection is free. Rendering a document from rows costs more on the read than +reading a stored document, and in exchange it ends write amplification; storing the +document makes the read cheaper and the writes heavier. Every pattern below explores +where that trade flips. **Duality does not change the physics knobs. It changes how the +knobs get set.** + +The three knobs, on every design: + +1. **Diversity of access patterns**: how many consumers want a *different shape* of + the same truth? (High diversity wants projections: one stored shape can't serve + every consumer. One dominant shape favors storing that shape.) +2. **Read/write ratio**: *where* compute should happen, write-time or read-time. +3. **Update %, and where the update lands**: the knob that quietly kills more designs + than the other two combined. A high-velocity field buried in a large document is + write amplification waiting to happen. + +Knob 3 can veto the other two: a read-heavy, high-diversity workload still can't embed +a field that is hot and mutable. + +The breakpoint to keep in your head: **maintain a precomputed structure only if +`read-freq × read-cost > write-freq × maintenance-cost`.** Writes get heavy or reads +get rare, and it flips. Read-time compute is never free, but a real cost-based +optimizer lowers its cost, and that moves the breakpoint. + +--- + +## How this repo supports the presentation + +This repository is the **runnable backing** for the 90-minute lecture *"Model the +Domain, Not the Engine."* Nothing in the talk is a claim you have to take on faith: +every pattern, every needle-flip, and the "one truth, many shapes" thesis itself is a +command you can run here, live, on a laptop. It serves two audiences: + +- **Presenters**: step out of the slides and into a terminal at any pattern. + `./run.sh 03-bucket` proves the Bucket `$sql` rollup and its cross-API parity in + front of the room; the numbers on the slides are reproduced by the same scripts. +- **Developers you send here afterward**: the repo *is* the takeaway. Each pattern is + a side-by-side they can read, run, modify, and measure against their own cardinality. + +### Lecture → repo map + +| Lecture beat | What backs it, here | +|---|---| +| **Why converged / "model once, project many"**: one truth served as many shapes | The duality views (`xr_client_dv`, `cp_subscriber_dv`, `ol_advisor_dv`) read **identically** through SQL and the MongoDB API; the `03-parity.js` scripts assert it byte-for-byte | +| **The physics**: three knobs, write amplification, the breakpoint | *The one idea* (above); every `01-document-model.sql` makes the write cost explicit in comments, every `02-converged.sql` shows where the needle flips | +| **The pattern walk**: six patterns, six industries | `patterns/01…06`, **1:1 with the slides**: same patterns, same industries, same hard edges | +| **Bucket's `$sql`-in-pipeline value-add**: parallel analytics, no pipeline caps | `patterns/03-bucket/02-sql-in-pipeline.js`: full SQL over the Mongo wire, runnable | +| **"One truth, many shapes": proven, not asserted** | The four cross-API parity scripts (`0{1,2,6}/03-parity.js`, `03/03-parity.js`): SQL result and MongoDB result asserted equal | +| **The rest of the catalog**: optional vs. obsolete | *The rest of the catalog* (below) | +| **The honesty anchor**: where document still wins | *Where document still wins* (below): the single-collection guidance, in words and as a rule | +| **Measure, don't guess**: the bake-off | *Measure it, don't guess it* (below) + the reason-then-measure modeling skill | +| **The anonymized proof numbers** (60s→500ms, 29s→sub-400ms, 9–15×) | Cited in the per-pattern READMEs, tagged as anonymized field results | + +### Live-demo playbook + +```bash +./run.sh 01-extended-reference # ⭐ flagship: the SAME document, SQL == MongoDB, byte-for-byte +./run.sh 03-bucket # the $sql-in-pipeline showcase + rollup parity +./run.sh # the whole walk, both lanes → 19 passed, 0 failed +``` + +Run any of these live: each `[PASS]` line is a slide's claim, executed. + +--- + +## The six patterns (1:1 with the lecture) + +| # | Pattern | Industry | Where the needle flips | Converged softening | +|---|---|---|---|---| +| [01](patterns/01-extended-reference/) ⭐ | **Extended Reference** | Wealth mgmt | Advisor moves offices → fan-out update across 100Ks of embedded copies + update anomaly | **Project, don't copy**: advisor normalized once, duality projects it live | +| [02](patterns/02-computed/) | **Computed** | Telecom | Every CDR re-aggregates + rewrites the whole subscriber doc → write storm | Append-only rows + trigger-maintained summary (staleness 0); **Top-N 60s → 500ms** | +| [03](patterns/03-bucket/) | **Bucket** | Manufacturing / IoT | Each reading rewrites the whole, growing bucket; marches at the 16 MB ceiling | INTERVAL-partitioned rows + `GROUP BY` rollup; **29s → sub-400ms** | +| [04](patterns/04-subset/) | **Subset** | Insurance | Push-and-trim on every claim to serve a full read that hardly happens | One table + composite index; **query the hot slice** with `FETCH FIRST` | +| [05](patterns/05-tree-hierarchy/) | **Tree / Hierarchy** | Manufacturing BOM | Re-parent → rewrites every descendant's path; where-used a prefix can't express | Adjacency edges (reorg = one row) + `CONNECT BY` + `GRAPH_TABLE` | +| [06](patterns/06-outlier/) | **Outlier** | Financial | `hasExtras` + overflow + app branch = the 16 MB limit leaking into your code | No special doc: "just more rows, the optimizer plans for it" | + +Each folder holds a `README.md` (the teaching), `01-document-model.sql` (the starting +point, with the write cost made explicit in comments), and `02-converged.sql` (the +alternative, with the needle-flip explained). The four patterns that expose a duality +view (01, 02, 06) or a rollup (03) also carry a **MongoDB lane**: `*.js` scripts that +run through `mongosh` and assert cross-API parity (see below). + +Numbers above are **anonymized field results** from real engagements, cited as such. +Your mileage depends on scale, cardinality, and access mix. *A win at the wrong scale +is not a win.* Measure your own (see below). + +--- + +## Repository layout + +``` +converged-modeling-patterns/ +├── run.sh # one command: build, wait, grant, run every pattern × both lanes, graded +├── compose.yml # the 26ai + ORDS + Mongo-API service (non-default host ports) +├── lab.sh # the same stack with the plain podman/docker CLI, for hosts with no compose +├── docker/ # the image, built from scratch: DB + ORDS + mongosh, no ONNX/vector layer +│ ├── Dockerfile +│ ├── scripts/ # install-ords.sh, entrypoint.sh (ORDS-enable CMP_USER + mongo.enabled) +│ └── init/ # 01-grants.sql, 02-ords-enable.sql +├── app/ # the hands-on console: Node 22 Express app, compose service `lab-ui` on :3100 +│ ├── src/ # server, gate/cache, SQL+Mongo runners, content loader, HTTP + admin routes +│ ├── public/ # the browser UI (offline vendor bundle, no outbound requests) +│ └── test/ # unit, integration, smoke (Puppeteer) and load tests +└── patterns/ + ├── 01-extended-reference/ # ⭐ wealth mgmt, duality projection + │ ├── README.md # the teaching: use case → hard edge → needle-flip → softening + │ ├── 01-document-model.sql# the starting point (write cost made explicit in comments) + │ ├── 02-converged.sql # the converged alternative (the flip, explained) + │ ├── 03-parity.js # Mongo lane: SAME doc via SQL and MongoDB API, byte-equal + │ └── _capture.sql # captures the SQL-side result for the parity assertion + ├── 02-computed/ # telecom, rollup, staleness 0 (+ 03-parity.js, _capture.sql) + ├── 03-bucket/ # manufacturing/IoT, time-series + │ ├── 01-document-model.sql / .js # the $push bucket that grows (SQL + Mongo lanes) + │ ├── 02-converged.sql # partitioned rows + GROUP BY rollup + │ ├── 02-sql-in-pipeline.js# $sql over the Mongo wire (the value-add) + │ └── 03-parity.js # Mongo $sql rollup == SQL GROUP BY, asserted + ├── 04-subset/ # insurance, pure SQL (no natural single Mongo collection) + ├── 05-tree-hierarchy/ # manufacturing BOM, pure SQL (adjacency + CONNECT BY + GRAPH_TABLE) + └── 06-outlier/ # financial, duality projection (+ 03-parity.js, _capture.sql) +``` + +Every pattern folder is self-contained and teaches the same four beats as its slide. +`.sql` files run in the SQL lane; `.js` files run in the MongoDB lane; `03-parity.js` +is the cross-API assertion; `_capture.sql` feeds the SQL side of that assertion to +`run.sh`. Read a folder's `README.md` first, then the two model files side by side. + +--- + +## Quickstart + +```bash +./run.sh # build/bring up the stack, then run ALL patterns, BOTH lanes +./run.sh 03-bucket # run just one pattern (both lanes) +``` + +`run.sh` builds and starts the container via compose (or `./lab.sh` when no compose +provider is installed: see *Podman* below), waits for the database +**and** the MongoDB API, grants the schema privileges the patterns use, and then, per +pattern, runs the **SQL lane** (`sqlplus`) followed by the **MongoDB lane** (`mongosh`, +in-container). Every script is graded: any ORA-/PLS- error or any parity mismatch is a +non-zero exit reported as **FAIL**. The run ends with a combined `N passed, 0 failed`. +It is idempotent; rerun freely. SQL-only patterns (Subset, Tree) have no `*.js` and +simply skip the Mongo lane. + +The image is built fresh from [`docker/`](docker/): Oracle AI Database 26ai Free +(`gvenzl/oracle-free:23.26.3-faststart`) + **ORDS** (fronting the Database API and the +Oracle API for MongoDB) + **mongosh**. No vector/ONNX layer: these patterns use no +vectors. Ports (host → container), defaulted to **non-standard** values so nothing +collides with another Oracle/ORDS/Mongo stack: + +| Service | In container | Host default | Override | +|---|---|---|---| +| SQL*Net | 1521 | **1522** | `CMP_PORT` | +| ORDS / Database Actions | 8181 | **8182** | `CMP_ORDS_PORT` | +| Oracle API for MongoDB | 27017 | **27018** | `CMP_MONGO_PORT` | +| Hands-on console (`lab-ui`) | 3000 | **3100** | `CMP_UI_PORT` | + +All four bind to `127.0.0.1` on the host (`CMP_DB_BIND` / `CMP_UI_BIND` to change). + +Drive it yourself: + +```bash +docker compose up -d --build + +# SQL lane +docker exec -i cmp-oracle sqlplus -s cmp_user/CmpUser2026@localhost:1521/FREEPDB1 \ + < patterns/01-extended-reference/01-document-model.sql + +# MongoDB lane (in-container mongosh; TLS is off for local dev) +MURI='mongodb://cmp_user:CmpUser2026@localhost:27017/CMP_USER?authMechanism=PLAIN&authSource=$external&retryWrites=false&loadBalanced=true' +docker exec -i -e MURI="$MURI" cmp-oracle bash -lc 'mongosh "$MURI" --quiet --file /dev/stdin' \ + < patterns/03-bucket/02-sql-in-pipeline.js +``` + +Login: `cmp_user / CmpUser2026`, service `FREEPDB1`, schema ORDS-enabled as `CMP_USER`. +Override `ORACLE_PASSWORD` / `CMP_PASSWORD` as you like. + +### Podman + +Podman works with or without a compose provider. With `podman compose` (or +`docker-compose` installed as its provider) everything above works as written. Without +one, use `lab.sh`, which runs the same two services with the plain `podman` CLI: + +```bash +./lab.sh up # build images if missing (--build to force), start the database, then the console +./lab.sh status # containers, health, console URL +./lab.sh logs oracle # or: lab-ui +./lab.sh down # remove the containers; -v also deletes the database volume +``` + +`lab.sh` reads `.env` like compose does (a variable set in the shell wins) and mirrors +`compose.yml`'s images, container names, ports and volume, so the database survives a +switch between the two. `run.sh` picks compose when a provider answers and falls back +to `lab.sh` otherwise, so `./run.sh` works unchanged. The image builds on x86_64 and on +Apple silicon (arm64). `docker exec` in the commands above becomes `podman exec`. + +## Hands-on console + +`docker compose up -d` (or `./lab.sh up`) also starts **`lab-ui`** (a browser console for this repo) at +**http://localhost:3100**. Every query in the six patterns, and in the lecture, can be +read, copied, edited and run from there, against the same 26ai container `run.sh` uses: + +- one page per pattern: the problem, the three knob settings, and query cards for the + document model, the converged model and (where it exists) the MongoDB API; +- **Copy**, **Load into console**, **Run** on every card, plus a console pane with its + own **SQL** and **MongoDB** tabs for ad hoc statements; +- **Measure it** runs the document-model write and its converged counterpart back to + back in one exclusive slot. Each side runs once unmeasured as a warm-up (parse and + first-touch effects stay out of the numbers), then once measured: it reads the + engine's own statistics (redo size, block changes, logical reads) before and after, + with in-memory undo switched off so the counters are current, and rolls back. Numbers are a + single-session, 26ai Free measurement: expect small run-to-run variance, not a fixed + constant; +- **Reset this pattern** rebuilds its tables to the starting state. + +The console lives in a **dock** collapsed to a thin bar at the bottom of the page. Hover +the bar, or use **Load into console**, to open it. It stays open while you're typing in +it or a console run is in progress, and closes on its own about half a second after both +your pointer leaves it and focus moves elsewhere (e.g. clicking back into the page). +Click **📌** to pin it open regardless, or **▲/▼ Console** to toggle it by hand. + +Every card, tab and knob has a **ⓘ** next to it: hover for a quick tooltip, click for a +dialog with what it does, why it matters and what to look for, and, where relevant, the +deck figure it illustrates. + +**Measure it**'s chart plots a reference line (write amplification measured on 26ai Free +at several sizes, same protocol) against **your** live point at this lab's actual size (on fresh or reset lab data), +so you can see where your result falls on the curve rather than judging it in isolation. +Beside the chart, a **deck workload model** thumbnail opens the pattern's full daily-cost +illustration from the deck; it's a separate, illustrative model, not something Measure it +proves directly. + +Execution is serialized: one statement runs against the database at a time, a +first-come-first-served queue shows your place ("queued · N ahead"), and repeated +read-only queries are served from a cache, so the console stays responsive with a full +room on the 26ai Free container's 2 CPU threads / 2 GB RAM. + +| Setting | Default | Meaning | +|---|---|---| +| `LAB_MODE` | `solo` | `solo`: one user, no sign-in. `event`: sign-in, a private workspace per attendee, admin page | +| `CMP_UI_PORT` | `3100` | host port for the console | +| `CMP_UI_BIND` | `127.0.0.1` | interface the console binds to; `0.0.0.0` for an event so the room can reach it | +| `CMP_DB_BIND` | `127.0.0.1` | interface for the database ports (1522/8182/27018); keep it on localhost, even for an event | +| `LAB_ADMIN_PASSWORD` | `LabAdmin2026` | the console's provisioning account; event mode refuses to start on the default | +| `DB_POOL_MAX` / `MONGO_POOL_MAX` | `1` / `1` | connection caps behind the queue | +| `EVENT_CODE`, `ADMIN_PASSWORD` | none | event mode only; see [`docs/instructor-runbook.md`](docs/instructor-runbook.md) | + +Every host port binds to `127.0.0.1` by default. For an event, publish only the console +(`CMP_UI_BIND=0.0.0.0`): publishing the database ports would hand attendees LAB_ADMIN and +SYS, whose credentials are in this repo. Don't run `./run.sh` during an event: its test +stage shares the database. + +Solo mode (the default) needs no configuration. `./run.sh` folds the console's own unit +and integration tests in as a last stage against the same database: **19 passed, 0 +failed**, combined with the patterns. `cd app && npm run test:smoke` is a separate +headless-Chrome walkthrough of every pattern page (screenshots land in +`app/test/artifacts/`, git-ignored), and `npm run test:load` drives a simulated event; +see [`docs/instructor-runbook.md`](docs/instructor-runbook.md) for running it against +your own room size before a large session. + +## The MongoDB lane + cross-API parity + +The Oracle API for MongoDB surfaces the same schema over the Mongo wire protocol, so a +MongoDB developer meets these patterns in their own tools, and the repo proves the two +lanes agree: + +- **`$sql`-in-pipeline** (Bucket): `patterns/03-bucket/02-sql-in-pipeline.js` issues the + hourly time-series rollup as **full SQL through the Mongo wire** via Oracle's `$sql` + aggregation stage: parallel execution, cost-based optimization, and none of the + 100 MB stage / 16 MB output caps a native pipeline hits. +- **Rollup parity** (Bucket): `03-parity.js` asserts the Mongo `$sql` rollup equals the + SQL lane's `02-converged.sql` `GROUP BY` (same machine/hour COUNT/AVG/MAX), and that + both equal the hand-maintained document-model bucket counters. +- **Document parity** (Extended Reference ⭐, Computed, Outlier): each `03-parity.js` + reads the **same duality-view document two ways**: `SELECT ... FROM ` (SQL) and + `db..findOne(...)` (MongoDB API), and asserts they are **byte-equal** after + canonicalizing JSON and ignoring the duality `_metadata` (etag/asof). This is the + executable proof of *"one truth, many shapes."* + +Each parity script `quit(1)`s on any mismatch, so `run.sh` reports it as FAIL. Subset +and Tree stay pure-SQL (no natural single Mongo collection for a claims table or an +adjacency/graph traversal). + +--- + +## The rest of the catalog + +The six patterns above are the ones where the tradeoff is closest. The rest of the +document pattern catalog splits into two piles on a converged engine: + +- **Still useful, now optional**: *Polymorphic* (differently shaped documents in one + collection, spanned by a multivalue index), *Attribute* (key/value pairs get a real + index instead of a scan), *Schema Versioning* (an `IS JSON` check plus a + discriminator, with a view presenting one shape), *Document Versioning* (temporal + validity and Flashback give the history without a second collection of copies). +- **Obsolete**: *Approximation* (native `APPROX_COUNT_DISTINCT` and the other + approximate aggregates answer directly), *Pre-Allocation* (padding documents so they + would not move on disk was an artifact of one early storage engine; when the engine + changed, the pattern vanished). + +Patterns are bets against an engine's limits, not laws of modeling. + +--- + +## Where document still wins (the honesty anchor) + +This is not "relational beats document." It is *shape per access pattern, on one +engine.* The document model is the right tool for most reads, and for a real class of +writes. The white-hot case (a shopping cart taking tens of millions of tiny add/remove +operations a day and read constantly) belongs in a **single collection**: on 26ai, a +JSON collection table (`CREATE JSON COLLECTION TABLE carts`) holding each cart as one +self-contained document, written through the MongoDB API and readable with SQL/JSON. +Same table, two access surfaces, one engine. + +Its knobs say embed. **Diversity** is low (one owner, one screen); **reads and writes** +are both hot, so there is nothing to precompute; **update locality** is private (an +update touches one cart, never a shared fact). Nothing is shared and nothing is +copied, so embedding costs nothing extra. Same three knobs as Extended Reference, +different settings, opposite answer. + +**Joins are never free.** One indexed read of a self-contained document is one B-tree +probe plus one fetch, O(log n). A duality view assembles the same shape from k tables: +roughly O(k · log n), and nested arrays cost more again. For the advisor card in +pattern 01 that premium buys atomic reorgs; for the cart it buys nothing, because +there is no shared fact to protect. **A single collection is not a duality view**: a +duality view assembles a document from tables; a single collection stores the document. +Both are first-class here. + +The rule: reach for a projection when a fact is **shared, queried many ways, or changes +under you**. Leave it embedded when it is **private, uniform and self-contained**. + +The other half of the 90/10: entitlement / array-containment workloads have run +**~9–15× faster in the document shape** than a 27-table normalized schema, on the +*same* engine. The point was never the shape. The point is you no longer have to pick +one for the whole system. + +--- + +## Measure it, don't guess it + +Every needle-flip in this repo is a *breakpoint*: a frequency or scale at which the +winner changes. On a converged engine you don't have to guess where it is: +reproduce the schema, generate data at real cardinality, build the candidate shapes, +and **measure read _and_ write cost** (writes are access patterns too). See the +`oracle-data-modeling` skill's reason-then-measure bake-off, and bring your own schema +and slowest queries to the workshop lab. + +--- + +## Notes + +- **26ai throughout.** Validated on Oracle AI Database 26ai Free (`23.26.3-faststart`), + ORDS 26.2, mongosh from the current MongoDB 8.0 repo. Every `.sql`, every `.js`, and + every cross-API parity assertion runs clean on a live container, and `./run.sh` folds + in the hands-on console's own test suite as a last stage (`19 passed, 0 failed`). +- **Customer-neutral.** No named customers; the domains are illustrative and reusable. +- **Two lanes, asserted equal.** The SQL lane validates via `sqlplus`; the MongoDB lane + validates via `mongosh`, and the parity scripts assert the two lanes return identical + data. See *The MongoDB lane + cross-API parity* above. + +## License + +Universal Permissive License (UPL) 1.0, see [LICENSE.txt](../../LICENSE.txt). Copyright (c) 2026 +Oracle and/or its affiliates. diff --git a/labs/converged-modeling-patterns/app/.gitignore b/labs/converged-modeling-patterns/app/.gitignore new file mode 100644 index 0000000..8dbf02e --- /dev/null +++ b/labs/converged-modeling-patterns/app/.gitignore @@ -0,0 +1,3 @@ +node_modules/ +public/vendor/ +test/artifacts/ diff --git a/labs/converged-modeling-patterns/app/Dockerfile b/labs/converged-modeling-patterns/app/Dockerfile new file mode 100644 index 0000000..c110327 --- /dev/null +++ b/labs/converged-modeling-patterns/app/Dockerfile @@ -0,0 +1,26 @@ +# app/Dockerfile +FROM node:22-bookworm-slim AS deps +WORKDIR /app +COPY app/package.json app/package-lock.json ./ +RUN npm ci +COPY app/ ./ +RUN npm run build:vendor + +FROM deps AS test +COPY patterns /patterns +ENV PATTERNS_DIR=/patterns +CMD ["npx", "vitest", "run"] + +FROM node:22-bookworm-slim AS runtime +WORKDIR /app +ENV NODE_ENV=production +COPY app/package.json app/package-lock.json ./ +RUN npm ci --omit=dev +COPY --from=deps /app/src ./src +COPY --from=deps /app/sql ./sql +COPY --from=deps /app/public ./public +COPY patterns /patterns +ENV PATTERNS_DIR=/patterns PORT=3000 +EXPOSE 3000 +USER node +CMD ["node", "src/server.js"] diff --git a/labs/converged-modeling-patterns/app/build/cm-entry.js b/labs/converged-modeling-patterns/app/build/cm-entry.js new file mode 100644 index 0000000..9e4cf43 --- /dev/null +++ b/labs/converged-modeling-patterns/app/build/cm-entry.js @@ -0,0 +1,47 @@ +// app/build/cm-entry.js +import { EditorView, basicSetup } from 'codemirror'; +import { keymap } from '@codemirror/view'; +import { Prec } from '@codemirror/state'; +import { sql, PLSQL } from '@codemirror/lang-sql'; +import { javascript } from '@codemirror/lang-javascript'; +import { HighlightStyle, syntaxHighlighting } from '@codemirror/language'; +import { tags as t } from '@lezer/highlight'; + +// Colours come from lab.css tokens, so the editor follows the page's light/dark scheme +// (CodeMirror's default highlight style is light-only and unreadable on the dark panel). +const labHighlight = HighlightStyle.define([ + { tag: [t.keyword, t.operatorKeyword, t.modifier, t.controlKeyword], color: 'var(--flow)' }, + { tag: [t.string, t.special(t.string), t.regexp], color: 'var(--hot)' }, + { tag: [t.number, t.bool, t.null, t.atom], color: 'var(--cool)' }, + { tag: [t.comment, t.lineComment, t.blockComment], color: 'var(--faint)', fontStyle: 'italic' }, + { tag: [t.typeName, t.standard(t.name)], color: 'var(--cool)' }, +]); +const labTheme = EditorView.theme({ + '&': { color: 'var(--ink)', backgroundColor: 'var(--panel)' }, + '.cm-content': { caretColor: 'var(--ink)' }, + '.cm-cursor, .cm-dropCursor': { borderLeftColor: 'var(--ink)' }, + '.cm-gutters': { backgroundColor: 'var(--paper)', color: 'var(--faint)', borderRight: '1px solid var(--hairline)' }, + '.cm-activeLine': { backgroundColor: 'var(--flow-soft)' }, + '.cm-activeLineGutter': { backgroundColor: 'var(--chip)', color: 'var(--ink)' }, + '&.cm-focused > .cm-scroller > .cm-selectionLayer .cm-selectionBackground, .cm-selectionBackground': { backgroundColor: 'var(--flow-soft)' }, +}); + +export function createEditor(parent, { lang, doc = '', onRun }) { + const view = new EditorView({ + parent, + doc, + extensions: [ + basicSetup, + labTheme, + syntaxHighlighting(labHighlight), + lang === 'sql' ? sql({ dialect: PLSQL }) : javascript(), + Prec.highest(keymap.of([{ key: 'Mod-Enter', run: () => { onRun?.(); return true; } }])), + EditorView.lineWrapping, + ], + }); + return { + get: () => view.state.doc.toString(), + set: (text) => view.dispatch({ changes: { from: 0, to: view.state.doc.length, insert: text } }), + focus: () => view.focus(), + }; +} diff --git a/labs/converged-modeling-patterns/app/build/vendor.mjs b/labs/converged-modeling-patterns/app/build/vendor.mjs new file mode 100644 index 0000000..3fa9f85 --- /dev/null +++ b/labs/converged-modeling-patterns/app/build/vendor.mjs @@ -0,0 +1,28 @@ +// app/build/vendor.mjs +import fs from 'node:fs'; +import path from 'node:path'; +import { build } from 'esbuild'; + +const root = path.resolve(import.meta.dirname, '..'); +const out = path.join(root, 'public/vendor'); +fs.mkdirSync(path.join(out, 'fonts'), { recursive: true }); + +await build({ entryPoints: [path.join(root, 'build/cm-entry.js')], bundle: true, format: 'esm', minify: true, outfile: path.join(out, 'cm.js') }); + +const FONTS = [ + ['bricolage-grotesque', 'Bricolage Grotesque', [600, 700, 800]], + ['public-sans', 'Public Sans', [400, 500, 600, 700]], + ['ibm-plex-mono', 'IBM Plex Mono', [400, 500, 600]], +]; +let css = ''; +for (const [pkg, family, weights] of FONTS) { + for (const w of weights) { + const file = `${pkg}-latin-${w}-normal.woff2`; + const src = path.join(root, 'node_modules/@fontsource', pkg, 'files', file); + if (!fs.existsSync(src)) throw new Error(`missing font file ${src}`); + fs.copyFileSync(src, path.join(out, 'fonts', file)); + css += `@font-face{font-family:'${family}';font-style:normal;font-weight:${w};font-display:swap;src:url(fonts/${file}) format('woff2');}\n`; + } +} +fs.writeFileSync(path.join(out, 'fonts.css'), css); +console.log('vendor: cm.js + fonts.css written'); diff --git a/labs/converged-modeling-patterns/app/package-lock.json b/labs/converged-modeling-patterns/app/package-lock.json new file mode 100644 index 0000000..37e4a41 --- /dev/null +++ b/labs/converged-modeling-patterns/app/package-lock.json @@ -0,0 +1,4062 @@ +{ + "name": "cmp-lab-ui", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "cmp-lab-ui", + "dependencies": { + "acorn": "^8.18.0", + "express": "^5.2.1", + "mongodb": "^7.7.0", + "oracledb": "^7.0.1", + "yaml": "^2.9.1" + }, + "devDependencies": { + "@codemirror/lang-javascript": "^6.2.5", + "@codemirror/lang-sql": "^6.10.0", + "@codemirror/language": "^6.12.4", + "@codemirror/state": "^6.7.6", + "@codemirror/view": "^6.43.13", + "@fontsource/bricolage-grotesque": "^5.3.0", + "@fontsource/ibm-plex-mono": "^5.3.0", + "@fontsource/public-sans": "^5.3.0", + "@lezer/highlight": "^1.2.5", + "codemirror": "^6.0.2", + "esbuild": "^0.28.2", + "jsdom": "^29.1.1", + "puppeteer-core": "^25.12.0", + "supertest": "^7.3.0", + "vitest": "^5.0.2" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/@asamuzakjp/css-color": { + "version": "5.1.11", + "resolved": "https://registry.npmjs.org/@asamuzakjp/css-color/-/css-color-5.1.11.tgz", + "integrity": "sha512-KVw6qIiCTUQhByfTd78h2yD1/00waTmm9uy/R7Ck/ctUyAPj+AEDLkQIdJW0T8+qGgj3j5bpNKK7Q3G+LedJWg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@asamuzakjp/generational-cache": "^1.0.1", + "@csstools/css-calc": "^3.2.0", + "@csstools/css-color-parser": "^4.1.0", + "@csstools/css-parser-algorithms": "^4.0.0", + "@csstools/css-tokenizer": "^4.0.0" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/@asamuzakjp/dom-selector": { + "version": "7.1.1", + "resolved": "https://registry.npmjs.org/@asamuzakjp/dom-selector/-/dom-selector-7.1.1.tgz", + "integrity": "sha512-67RZDnYRc8H/8MLDgQCDE//zoqVFwajkepHZgmXrbwybzXOEwOWGPYGmALYl9J2DOLfFPPs6kKCqmbzV895hTQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@asamuzakjp/generational-cache": "^1.0.1", + "@asamuzakjp/nwsapi": "^2.3.9", + "bidi-js": "^1.0.3", + "css-tree": "^3.2.1", + "is-potential-custom-element-name": "^1.0.1" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/@asamuzakjp/generational-cache": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/@asamuzakjp/generational-cache/-/generational-cache-1.0.1.tgz", + "integrity": "sha512-wajfB8KqzMCN2KGNFdLkReeHncd0AslUSrvHVvvYWuU8ghncRJoA50kT3zP9MVL0+9g4/67H+cdvBskj9THPzg==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/@asamuzakjp/nwsapi": { + "version": "2.3.9", + "resolved": "https://registry.npmjs.org/@asamuzakjp/nwsapi/-/nwsapi-2.3.9.tgz", + "integrity": "sha512-n8GuYSrI9bF7FFZ/SjhwevlHc8xaVlb/7HmHelnc/PZXBD2ZR49NnN9sMMuDdEGPeeRQ5d0hqlSlEpgCX3Wl0Q==", + "dev": true, + "license": "MIT" + }, + "node_modules/@bramus/specificity": { + "version": "2.4.2", + "resolved": "https://registry.npmjs.org/@bramus/specificity/-/specificity-2.4.2.tgz", + "integrity": "sha512-ctxtJ/eA+t+6q2++vj5j7FYX3nRu311q1wfYH3xjlLOsczhlhxAg2FWNUXhpGvAw3BWo1xBcvOV6/YLc2r5FJw==", + "dev": true, + "license": "MIT", + "dependencies": { + "css-tree": "^3.0.0" + }, + "bin": { + "specificity": "bin/cli.js" + } + }, + "node_modules/@codemirror/autocomplete": { + "version": "6.20.3", + "resolved": "https://registry.npmjs.org/@codemirror/autocomplete/-/autocomplete-6.20.3.tgz", + "integrity": "sha512-tlosUqb+3BbxCxZdu4tKeRghPFC+QM7q4X5YhKV2eCmPG+1r2F3f4AaSz5sCrFqUtX4Jh20VFTKecl16MgiV9g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@codemirror/language": "^6.0.0", + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.17.0", + "@lezer/common": "^1.0.0" + } + }, + "node_modules/@codemirror/commands": { + "version": "6.11.1", + "resolved": "https://registry.npmjs.org/@codemirror/commands/-/commands-6.11.1.tgz", + "integrity": "sha512-O/4hG3SC1YwcmQ0d2UVNDs+AsaNWd1iHVxbTeEBuqH+6bExAiPK3iS/BvpY6rZGURALv4ZD3sIgcCmRvw3ehBg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@codemirror/language": "^6.0.0", + "@codemirror/state": "^6.7.0", + "@codemirror/view": "^6.27.0", + "@lezer/common": "^1.1.0" + } + }, + "node_modules/@codemirror/lang-javascript": { + "version": "6.2.5", + "resolved": "https://registry.npmjs.org/@codemirror/lang-javascript/-/lang-javascript-6.2.5.tgz", + "integrity": "sha512-zD4e5mS+50htS7F+TYjBPsiIFGanfVqg4HyUz6WNFikgOPf2BgKlx+TQedI1w6n/IqRBVBbBWmGFdLB/7uxO4A==", + "dev": true, + "license": "MIT", + "dependencies": { + "@codemirror/autocomplete": "^6.0.0", + "@codemirror/language": "^6.6.0", + "@codemirror/lint": "^6.0.0", + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.17.0", + "@lezer/common": "^1.0.0", + "@lezer/javascript": "^1.0.0" + } + }, + "node_modules/@codemirror/lang-sql": { + "version": "6.10.0", + "resolved": "https://registry.npmjs.org/@codemirror/lang-sql/-/lang-sql-6.10.0.tgz", + "integrity": "sha512-6ayPkEd/yRw0XKBx5uAiToSgGECo/GY2NoJIHXIIQh1EVwLuKoU8BP/qK0qH5NLXAbtJRLuT73hx7P9X34iO4w==", + "dev": true, + "license": "MIT", + "dependencies": { + "@codemirror/autocomplete": "^6.0.0", + "@codemirror/language": "^6.0.0", + "@codemirror/state": "^6.0.0", + "@lezer/common": "^1.2.0", + "@lezer/highlight": "^1.0.0", + "@lezer/lr": "^1.0.0" + } + }, + "node_modules/@codemirror/language": { + "version": "6.12.4", + "resolved": "https://registry.npmjs.org/@codemirror/language/-/language-6.12.4.tgz", + "integrity": "sha512-1q4PaT+o6PbgpkJt4Q8Fv5XJxTy4FUZ4MWETtyiDw3J0Pyr9E2vqcKL+k9wcvjNTIsauxvE7OfmWj3FRPHQ76A==", + "dev": true, + "license": "MIT", + "dependencies": { + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.23.0", + "@lezer/common": "^1.5.0", + "@lezer/highlight": "^1.0.0", + "@lezer/lr": "^1.0.0", + "style-mod": "^4.0.0" + } + }, + "node_modules/@codemirror/lint": { + "version": "6.9.7", + "resolved": "https://registry.npmjs.org/@codemirror/lint/-/lint-6.9.7.tgz", + "integrity": "sha512-28/+iWLYxKxsvGYhSYL7zaCZqLz5+FFFDq9tVsvGv9kv8RY4fFAchJ5WX9M3YrrRlTIsECjsXPqeNgnSmNP2dg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.42.0", + "crelt": "^1.0.5" + } + }, + "node_modules/@codemirror/search": { + "version": "6.7.2", + "resolved": "https://registry.npmjs.org/@codemirror/search/-/search-6.7.2.tgz", + "integrity": "sha512-gUYkYhT2+n/+VGZ+8EzE5WFkYZUZYm1VOKDudIsNqh42uRVQJ0a6Yss9sdKT3MeOYfuL1N6AZA57oza0Oyr0LA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.37.0", + "crelt": "^1.0.5" + } + }, + "node_modules/@codemirror/state": { + "version": "6.7.6", + "resolved": "https://registry.npmjs.org/@codemirror/state/-/state-6.7.6.tgz", + "integrity": "sha512-kAz+AncRtKuIknedxT1bq4XwXv4UowhbkHU1myPrtVb/jZtImWuV5BXzv5vK6i3kYACsdiZiQKFQQ5Mq7elW8w==", + "dev": true, + "license": "MIT", + "dependencies": { + "@marijn/find-cluster-break": "^1.0.0" + } + }, + "node_modules/@codemirror/view": { + "version": "6.43.13", + "resolved": "https://registry.npmjs.org/@codemirror/view/-/view-6.43.13.tgz", + "integrity": "sha512-sihaFrUzAsYBQsL9J2t69y8nfMQGwcYmggAZsk+kjPbjYZMyuf2hU8tUNTZ+P+isb6XRr8JE22TZlJxBoVdH1A==", + "dev": true, + "license": "MIT", + "dependencies": { + "@codemirror/state": "^6.7.0", + "crelt": "^1.0.6", + "style-mod": "^4.1.0", + "w3c-keyname": "^2.2.4" + } + }, + "node_modules/@csstools/color-helpers": { + "version": "6.1.2", + "resolved": "https://registry.npmjs.org/@csstools/color-helpers/-/color-helpers-6.1.2.tgz", + "integrity": "sha512-grhRy3OKmniaAEKXMjua5z/EODX0MSqBGjunw8+j/3HQjOnahs2AGhvEOIYVUWcU6ScApbhLhVrQTX8XqrMrow==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT-0", + "engines": { + "node": ">=20.19.0" + } + }, + "node_modules/@csstools/css-calc": { + "version": "3.4.1", + "resolved": "https://registry.npmjs.org/@csstools/css-calc/-/css-calc-3.4.1.tgz", + "integrity": "sha512-EtC7SoN1j6J4E4DCwg5QgbO5TGxgxIA1RXqe+W+qUM+BUcezx9wT+/tiQ/WO2yCX4i5X+Cuf9ciJ22aP4UEwWw==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "engines": { + "node": ">=20.19.0" + }, + "peerDependencies": { + "@csstools/css-parser-algorithms": "^4.0.1", + "@csstools/css-tokenizer": "^4.0.2" + } + }, + "node_modules/@csstools/css-color-parser": { + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@csstools/css-color-parser/-/css-color-parser-4.2.4.tgz", + "integrity": "sha512-DyefytAZ735mX4Dq/WcDAXFtXhaEFvme0ZS9tVEBAc2whxUthXr0R0L2rmEPm59SrMBkGrFzQviBXMs/UtnABQ==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "dependencies": { + "@csstools/color-helpers": "^6.1.2", + "@csstools/css-calc": "^3.4.1" + }, + "engines": { + "node": ">=20.19.0" + }, + "peerDependencies": { + "@csstools/css-parser-algorithms": "^4.0.1", + "@csstools/css-tokenizer": "^4.0.2" + } + }, + "node_modules/@csstools/css-parser-algorithms": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/@csstools/css-parser-algorithms/-/css-parser-algorithms-4.0.1.tgz", + "integrity": "sha512-ShL8BqPfbKJrJiKFH0xBbN0i7Nrh9HXYRuF+pzyj93R/BL2YAsUxJeqANErzqM+0JGl7vjHp+3OIgd/DL69YIA==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "engines": { + "node": ">=20.19.0" + }, + "peerDependencies": { + "@csstools/css-tokenizer": "^4.0.2" + } + }, + "node_modules/@csstools/css-syntax-patches-for-csstree": { + "version": "1.1.14", + "resolved": "https://registry.npmjs.org/@csstools/css-syntax-patches-for-csstree/-/css-syntax-patches-for-csstree-1.1.14.tgz", + "integrity": "sha512-HpbVXyrofRXpHpgkNIjU/3EWR4WJvOkO3emNK/L6X/mTJU7bGUI3AkkpoTNXznQLp0KRjLHELTGeKI5dIkI9JQ==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT-0", + "peerDependencies": { + "css-tree": "^3.2.1" + }, + "peerDependenciesMeta": { + "css-tree": { + "optional": true + } + } + }, + "node_modules/@csstools/css-tokenizer": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/@csstools/css-tokenizer/-/css-tokenizer-4.0.2.tgz", + "integrity": "sha512-OoKoR0f76dCY666JlcbhmVTs2drYj1GUXZTYTcbUgJjh9Nv41aFfZ21bPQTERm5+L5cBDo466NltB2lplS5GBw==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "engines": { + "node": ">=20.19.0" + } + }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.28.2.tgz", + "integrity": "sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.28.2.tgz", + "integrity": "sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.28.2.tgz", + "integrity": "sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.28.2.tgz", + "integrity": "sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.28.2.tgz", + "integrity": "sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.28.2.tgz", + "integrity": "sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.28.2.tgz", + "integrity": "sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.28.2.tgz", + "integrity": "sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.28.2.tgz", + "integrity": "sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.28.2.tgz", + "integrity": "sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.28.2.tgz", + "integrity": "sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.28.2.tgz", + "integrity": "sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.28.2.tgz", + "integrity": "sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.28.2.tgz", + "integrity": "sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.28.2.tgz", + "integrity": "sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.28.2.tgz", + "integrity": "sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.28.2.tgz", + "integrity": "sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-arm64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.28.2.tgz", + "integrity": "sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.28.2.tgz", + "integrity": "sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-arm64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.28.2.tgz", + "integrity": "sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.28.2.tgz", + "integrity": "sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openharmony-arm64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.28.2.tgz", + "integrity": "sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.28.2.tgz", + "integrity": "sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.28.2.tgz", + "integrity": "sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.28.2.tgz", + "integrity": "sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.28.2.tgz", + "integrity": "sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@exodus/bytes": { + "version": "1.16.0", + "resolved": "https://registry.npmjs.org/@exodus/bytes/-/bytes-1.16.0.tgz", + "integrity": "sha512-IcpW84uEn3N7ETtNZMlxKhfl6Pec8rUNGOTBtWbK1FKhJxIFAptZyVrvVRVBimAJxJCgc3PxepxkdWWG4DVzfA==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + }, + "peerDependencies": { + "@noble/hashes": "^1.8.0 || ^2.0.0" + }, + "peerDependenciesMeta": { + "@noble/hashes": { + "optional": true + } + } + }, + "node_modules/@fontsource/bricolage-grotesque": { + "version": "5.3.0", + "resolved": "https://registry.npmjs.org/@fontsource/bricolage-grotesque/-/bricolage-grotesque-5.3.0.tgz", + "integrity": "sha512-MdOVb/5in11IfN/IQJOExMuu6AroCIQbirl0yX7NUvKk+h0HYIKfGaVAXRS/gCPil4aFxWhjZnOw28tUwv7tFw==", + "dev": true, + "license": "OFL-1.1", + "funding": { + "url": "https://github.com/sponsors/ayuhito" + } + }, + "node_modules/@fontsource/ibm-plex-mono": { + "version": "5.3.0", + "resolved": "https://registry.npmjs.org/@fontsource/ibm-plex-mono/-/ibm-plex-mono-5.3.0.tgz", + "integrity": "sha512-eTgnZjZEGk1QtD3ZstF+Vclo2HLAni8YMy34/DxllwZvyz1lR/1RF/xTiAquOBO7MvqBx8D2Ig2WCPMVfdZu7Q==", + "dev": true, + "license": "OFL-1.1", + "funding": { + "url": "https://github.com/sponsors/ayuhito" + } + }, + "node_modules/@fontsource/public-sans": { + "version": "5.3.0", + "resolved": "https://registry.npmjs.org/@fontsource/public-sans/-/public-sans-5.3.0.tgz", + "integrity": "sha512-kjODI0S3zdv0mBYCIQ8TbBayaiqszpc2UbhJiO3bjIqVVXzcWfHSt2o3WBCLOY3juaGaQoy4MoWCcgmfI5hCuA==", + "dev": true, + "license": "OFL-1.1", + "funding": { + "url": "https://github.com/sponsors/ayuhito" + } + }, + "node_modules/@jridgewell/resolve-uri": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", + "integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@jridgewell/sourcemap-codec": { + "version": "1.6.0", + "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.6.0.tgz", + "integrity": "sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==", + "dev": true, + "license": "MIT" + }, + "node_modules/@jridgewell/trace-mapping": { + "version": "0.3.31", + "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz", + "integrity": "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/resolve-uri": "^3.1.0", + "@jridgewell/sourcemap-codec": "^1.4.14" + } + }, + "node_modules/@lezer/common": { + "version": "1.5.3", + "resolved": "https://registry.npmjs.org/@lezer/common/-/common-1.5.3.tgz", + "integrity": "sha512-H0iErY4e43LpXbYDyBci5W4v/RwTgGV3YzYOlJPiGZ8RY8w51kVE+xLn1tg3Z6UmSfyETvtsUK3r7YsgDQEI7Q==", + "dev": true, + "license": "MIT" + }, + "node_modules/@lezer/highlight": { + "version": "1.2.5", + "resolved": "https://registry.npmjs.org/@lezer/highlight/-/highlight-1.2.5.tgz", + "integrity": "sha512-O1GMVKgtf5YspFaRzpmqkVgMtIk2HG9uMAbH6wdAIdNh3TVzCsch/+cRi+6l2//UGDKe6NB9BW3cvuR5ezMQ0w==", + "dev": true, + "license": "MIT", + "dependencies": { + "@lezer/common": "^1.3.0" + } + }, + "node_modules/@lezer/javascript": { + "version": "1.5.5", + "resolved": "https://registry.npmjs.org/@lezer/javascript/-/javascript-1.5.5.tgz", + "integrity": "sha512-sWg4yX1J6XW67AaAynVt0iwF0M5c+np36TEu+P2ifAJ8haRYvHnWDV28r1jdwnJehWCwXECutAUy56K4RBZIyg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@lezer/common": "^1.2.0", + "@lezer/highlight": "^1.1.3", + "@lezer/lr": "^1.3.0" + } + }, + "node_modules/@lezer/lr": { + "version": "1.4.10", + "resolved": "https://registry.npmjs.org/@lezer/lr/-/lr-1.4.10.tgz", + "integrity": "sha512-rnCpTIBafOx4mRp43xOxDJbFipJm/c0cia/V5TiGlhmMa+wsSdoGmUN3w5Bqrks/09Q/D4tNAmWaT8p6NRi77A==", + "dev": true, + "license": "MIT", + "dependencies": { + "@lezer/common": "^1.0.0" + } + }, + "node_modules/@marijn/find-cluster-break": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/@marijn/find-cluster-break/-/find-cluster-break-1.0.4.tgz", + "integrity": "sha512-Wy0V7+SGUjnF9/TkiM1hKVDPj7jKXduPNboMVtHTA8dySMURWqfg/JZ9E2Sq8JgSJmkl7k7Qe9FLeMSrSraWmQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/@mongodb-js/saslprep": { + "version": "1.5.4", + "resolved": "https://registry.npmjs.org/@mongodb-js/saslprep/-/saslprep-1.5.4.tgz", + "integrity": "sha512-05UC0jQsjKAOuXQ0H9Ud9vUTJpZIg+n/FinpR30tI5I8pY2inTfPOZ5OF/cg3Ce/N9MoD1xhRCeOsJtuTbFYlw==", + "license": "MIT", + "dependencies": { + "sparse-bitfield": "^3.0.3" + } + }, + "node_modules/@noble/hashes": { + "version": "1.8.0", + "resolved": "https://registry.npmjs.org/@noble/hashes/-/hashes-1.8.0.tgz", + "integrity": "sha512-jCs9ldd7NwzpgXDIf6P3+NrHh9/sD6CQdxHyjQI+h/6rDNo88ypBxxz45UDuZHz9r3tNz7N/VInSVoVdtXEI4A==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^14.21.3 || >=16" + }, + "funding": { + "url": "https://paulmillr.com/funding/" + } + }, + "node_modules/@oxc-project/types": { + "version": "0.151.0", + "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.151.0.tgz", + "integrity": "sha512-J1yXrIlNDZVzE3ada310xeAw7nH8yCAyLPuUIsjKatFPmfn5bS1oW+cM+QsGOtVWd5nhSpbwZWx/rue+r5Z+PA==", + "dev": true, + "license": "MIT", + "peer": true, + "funding": { + "url": "https://github.com/sponsors/oxc-project" + } + }, + "node_modules/@paralleldrive/cuid2": { + "version": "2.3.1", + "resolved": "https://registry.npmjs.org/@paralleldrive/cuid2/-/cuid2-2.3.1.tgz", + "integrity": "sha512-XO7cAxhnTZl0Yggq6jOgjiOHhbgcO4NqFqwSmQpjK3b6TEE6Uj/jfSk6wzYyemh3+I0sHirKSetjQwn5cZktFw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@noble/hashes": "^1.1.5" + } + }, + "node_modules/@puppeteer/browsers": { + "version": "3.2.3", + "resolved": "https://registry.npmjs.org/@puppeteer/browsers/-/browsers-3.2.3.tgz", + "integrity": "sha512-2Bt3m6dDAJqmZehn0wiXSYyQmuhyiHHZ7FjvrVaS1W2RoFGV6eCK71UvKYkNJjkwC1Q9nobjhvTMhLQztFVINA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "modern-tar": "^0.8.4", + "yargs": "^18.0.0" + }, + "bin": { + "browsers": "lib/main-cli.js" + }, + "engines": { + "node": ">=22.12.0" + }, + "peerDependencies": { + "proxy-agent": ">=8.0.1", + "yauzl": "^2.10.0 || ^3.4.0" + }, + "peerDependenciesMeta": { + "proxy-agent": { + "optional": true + }, + "yauzl": { + "optional": true + } + } + }, + "node_modules/@rolldown/binding-android-arm-eabi": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm-eabi/-/binding-android-arm-eabi-1.2.11.tgz", + "integrity": "sha512-A5kXfGKvKWWZE0TtPrfsvT+q4Y5d1QG8gGUzpYjGydM+fARM9MuX90PrXYXe0XbsDVgyxxNzHo6giCj90bsFNw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-android-arm64": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.2.11.tgz", + "integrity": "sha512-z6cTycz+iJ4PVkuL4HHW4DfTfoeU/2nqYYuSOrTmH7yHK5Y0LCOnA03V4ZNxavyVaU1oOqUgIg2klN/s+USGOA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-darwin-arm64": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.2.11.tgz", + "integrity": "sha512-jShvqNtP6vDC6/A5JOAzbVV+DkgHqhl/ScVCJEbt+TUY6QYz7YnXcrg3sLtFBniro0f/Ld50ZwCWA6f7KYD1nQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-darwin-x64": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.2.11.tgz", + "integrity": "sha512-f2i2xiNWq1Z1l2++q2fuhZRdLAT3aqxD6vRNm1RAxpUoBcdqNB3C0s1Bt+K+PbEx2F5F4gQp6hqKkphCY/xF9w==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-freebsd-x64": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.2.11.tgz", + "integrity": "sha512-4Ir5FSOKIAMr4r0kExpt1s3bMgzJU3rA45AYOHtQpls0oNeqcYBKrWMlckrYH4KCfGLfkfn1tN1dmZPMVsdXow==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm-gnueabihf": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.2.11.tgz", + "integrity": "sha512-/gnRDM+39BROzAN/k1OZjDPnDMcZxB/0EUxKjONO5yVkNEvlsoMDrxGNKgZi/ttFriS2gwlDNzB65pvNbFOXIQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm64-gnu": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.2.11.tgz", + "integrity": "sha512-PFaK8HwvAHbaKbBcDNQihjMKYvFnA5hiENx/l5tphTDz1E0WFp32l0A7aq7lyUwGsRw/xSrNIy/gIK4thrSCrw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm64-musl": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.2.11.tgz", + "integrity": "sha512-AskzJUIKRLPxkruR1wLKewGbOw+EYfU/9lOrBFj4AFrEA8hPpKFnODWNu2WLaNs0QNkEb9QIJufmVZZIL/bJlg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-ppc64-gnu": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.2.11.tgz", + "integrity": "sha512-qlUGAheh2yh8afH7QBgx0PrRHN85hKnNd78x8MeMhXivuevgd8vgf6/CstOzmNKY/lLTHvNTrPy98cLnAugzJw==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-s390x-gnu": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.2.11.tgz", + "integrity": "sha512-secpEad+0vCbSfn8upFySkDskv+bGPk3THSDS9Y89yc4rb4kzqHp8Dmyd9BkQW4SnhNXBZCl/6CrO//hZahNJQ==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-x64-gnu": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.2.11.tgz", + "integrity": "sha512-mOVBT3dPpkWm8XBWPmU4bf+U6dYDLeMo/9ojUmis4N0L5uu10qra5vOyngZ7/PSdoE4G9KvRt4bloRxNjLas7A==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-x64-musl": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.2.11.tgz", + "integrity": "sha512-Is78i9A8Ui4SqcxUwFJ9uMmjDn58IbVTjFWYdQestFEgeuEmHMLGNriXnVJKkwG2YiZjw8cP0zCTyDMdDGtOOg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-openharmony-arm64": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.2.11.tgz", + "integrity": "sha512-dUCXneZ87INUMyQ0D+C0HrEBNUPNXHaPmU5GTjyKTJEiussw9Kaj5Ln8UztPe4epV/ffvgNBEadksdYhmW6xJA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-win32-arm64-msvc": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.2.11.tgz", + "integrity": "sha512-jByxb6qfd+bH1xUd0qnfFnb17i9sWBPY2tOavJ0l3tdr3OTu+Kvtm8cd/JV5nFt657b1VqGltxg9olOEfofXWw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-win32-x64-msvc": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.2.11.tgz", + "integrity": "sha512-/PzKqzAJ03i19oy2ItPvyvaVjOjBCNnfaJs8yvUdGBKmiESgnrJSQ2awd81QzFbbnAmu7YO9ZnJrDCb9VSJPRA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "peer": true, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/pluginutils": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.1.tgz", + "integrity": "sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw==", + "dev": true, + "license": "MIT", + "peer": true + }, + "node_modules/@types/chai": { + "version": "5.2.3", + "resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz", + "integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/deep-eql": "*", + "assertion-error": "^2.0.1" + } + }, + "node_modules/@types/deep-eql": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz", + "integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/estree": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz", + "integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/webidl-conversions": { + "version": "7.0.3", + "resolved": "https://registry.npmjs.org/@types/webidl-conversions/-/webidl-conversions-7.0.3.tgz", + "integrity": "sha512-CiJJvcRtIgzadHCYXw7dqEnMNRjhGZlYK05Mj9OyktqV8uVT8fD2BFOB7S1uwBE3Kj2Z+4UyPmFw/Ixgw/LAlA==", + "license": "MIT" + }, + "node_modules/@types/whatwg-url": { + "version": "13.0.0", + "resolved": "https://registry.npmjs.org/@types/whatwg-url/-/whatwg-url-13.0.0.tgz", + "integrity": "sha512-N8WXpbE6Wgri7KUSvrmQcqrMllKZ9uxkYWMt+mCSGwNc0Hsw9VQTW7ApqI4XNrx6/SaM2QQJCzMPDEXE058s+Q==", + "license": "MIT", + "dependencies": { + "@types/webidl-conversions": "*" + } + }, + "node_modules/@vitest/mocker": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-5.0.2.tgz", + "integrity": "sha512-Z5FS00Q1SJHkB35xATsmWGdQ5WA1/0MV3CDjqyv7GavHv1OfOj145MNfHOlHk7QLes21dKFDHr8EO2zvL+9WGA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/trace-mapping": "0.3.31", + "@vitest/spy": "5.0.2", + "estree-walker": "^3.0.3", + "magic-string": "^1.2.3" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "msw": "^2.4.9", + "vite": "^6.0.0 || ^7.0.0 || ^8.0.0" + }, + "peerDependenciesMeta": { + "msw": { + "optional": true + }, + "vite": { + "optional": true + } + } + }, + "node_modules/@vitest/spy": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-5.0.2.tgz", + "integrity": "sha512-Ijc7T1nT9efNb5LxvjaBrEqw3f/QwUv5EE0nKqZxgqsaV/FxAAZ8baGylA8X/Z2oS4Lp+K74Jr6dTJsDKxJDeg==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/accepts": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/accepts/-/accepts-2.0.0.tgz", + "integrity": "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng==", + "license": "MIT", + "dependencies": { + "mime-types": "^3.0.0", + "negotiator": "^1.0.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/acorn": { + "version": "8.18.0", + "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.18.0.tgz", + "integrity": "sha512-lGq+9yr1/GuAWaVYIHRjvvySG5/4VfKIvC8EWxStPdcDh/Ka7FG3twP6v4d5BkravUilhIAsG4Qj83t02LWUPQ==", + "license": "MIT", + "bin": { + "acorn": "bin/acorn" + }, + "engines": { + "node": ">=0.4.0" + } + }, + "node_modules/ansi-regex": { + "version": "6.4.0", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-6.4.0.tgz", + "integrity": "sha512-KzTVk2tCWAHtYrvvvaP8bJKJq2pVinhLcGEQdtLIYPbmNGNyYe8QwNaTUYQp2J7/vIsUKt5QCqAfUkYyG9DkOw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/chalk/ansi-regex?sponsor=1" + } + }, + "node_modules/ansi-styles": { + "version": "6.2.3", + "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-6.2.3.tgz", + "integrity": "sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/chalk/ansi-styles?sponsor=1" + } + }, + "node_modules/asap": { + "version": "2.0.6", + "resolved": "https://registry.npmjs.org/asap/-/asap-2.0.6.tgz", + "integrity": "sha512-BSHWgDSAiKs50o2Re8ppvp3seVHXSRM44cdSsT9FfNEUUZLOGWVCsiWaRPWM1Znn+mqZ1OfVZ3z3DWEzSp7hRA==", + "dev": true, + "license": "MIT" + }, + "node_modules/assertion-error": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz", + "integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + } + }, + "node_modules/asynckit": { + "version": "0.4.0", + "resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz", + "integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==", + "dev": true, + "license": "MIT" + }, + "node_modules/bidi-js": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/bidi-js/-/bidi-js-1.1.0.tgz", + "integrity": "sha512-fX1Onk0tdVPC7obPWB5EbJ1z7NVhLq4m2xZLq2YXBkxzMXIGRpNMU88n0EPgWseKl12J7zXs7qrDxPK4sRs2fg==", + "dev": true, + "license": "MIT", + "dependencies": { + "require-from-string": "^2.0.2" + } + }, + "node_modules/body-parser": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.3.0.tgz", + "integrity": "sha512-2cGmJupaNgg+QUwVLAucDuWuoMZ6EX9iHDRswZ5lsNYEmwPaRknMPCLZz07yTzVq/83p4o/wzbDZbBrTvGGTIw==", + "license": "MIT", + "dependencies": { + "bytes": "^3.1.2", + "content-type": "^2.0.0", + "debug": "^4.4.3", + "http-errors": "^2.0.1", + "iconv-lite": "^0.7.2", + "on-finished": "^2.4.1", + "qs": "^6.15.2", + "raw-body": "^3.0.2", + "type-is": "^2.1.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/body-parser/node_modules/content-type": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz", + "integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/bson": { + "version": "7.3.3", + "resolved": "https://registry.npmjs.org/bson/-/bson-7.3.3.tgz", + "integrity": "sha512-oz3LwE3qGEhTCSyhzQRLY3Bj1NLOsjFMf6tcmY5J8M+3R7EVcmoBfSHWvjUK3yXCLDUsGpIuB3aQL4nDZ1EV/g==", + "license": "Apache-2.0", + "engines": { + "node": ">=20.19.0" + } + }, + "node_modules/bytes": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz", + "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/call-bind-apply-helpers": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", + "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/call-bound": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz", + "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "get-intrinsic": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/chai": { + "version": "6.2.2", + "resolved": "https://registry.npmjs.org/chai/-/chai-6.2.2.tgz", + "integrity": "sha512-NUPRluOfOiTKBKvWPtSD4PhFvWCqOi0BGStNWs57X9js7XGTprSmFoz5F0tWhR4WPjNeR9jXqdC7/UpSJTnlRg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/chromium-bidi": { + "version": "17.0.2", + "resolved": "https://registry.npmjs.org/chromium-bidi/-/chromium-bidi-17.0.2.tgz", + "integrity": "sha512-5v9GQFhTktFvotn/OFNJBmKLKRAb6n9r0bVCwf7sHgWc3/JryK0bj1nn93L3pHFrfgcsu6Be6EWsDi+1XHTGDg==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "mitt": "^3.0.1", + "zod": "^3.24.1" + }, + "engines": { + "node": ">=20.19.0 <22.0.0 || >=22.12.0" + }, + "peerDependencies": { + "devtools-protocol": "*" + } + }, + "node_modules/cliui": { + "version": "9.0.1", + "resolved": "https://registry.npmjs.org/cliui/-/cliui-9.0.1.tgz", + "integrity": "sha512-k7ndgKhwoQveBL+/1tqGJYNz097I7WOvwbmmU2AR5+magtbjPWQTS1C5vzGkBC8Ym8UWRzfKUzUUqFLypY4Q+w==", + "dev": true, + "license": "ISC", + "dependencies": { + "string-width": "^7.2.0", + "strip-ansi": "^7.1.0", + "wrap-ansi": "^9.0.0" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/cliui/node_modules/string-width": { + "version": "7.2.0", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-7.2.0.tgz", + "integrity": "sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "emoji-regex": "^10.3.0", + "get-east-asian-width": "^1.0.0", + "strip-ansi": "^7.1.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/codemirror": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/codemirror/-/codemirror-6.0.2.tgz", + "integrity": "sha512-VhydHotNW5w1UGK0Qj96BwSk/Zqbp9WbnyK2W/eVMv4QyF41INRGpjUhFJY7/uDNuudSc33a/PKr4iDqRduvHw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@codemirror/autocomplete": "^6.0.0", + "@codemirror/commands": "^6.0.0", + "@codemirror/language": "^6.0.0", + "@codemirror/lint": "^6.0.0", + "@codemirror/search": "^6.0.0", + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.0.0" + } + }, + "node_modules/combined-stream": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz", + "integrity": "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==", + "dev": true, + "license": "MIT", + "dependencies": { + "delayed-stream": "~1.0.0" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/component-emitter": { + "version": "1.3.1", + "resolved": "https://registry.npmjs.org/component-emitter/-/component-emitter-1.3.1.tgz", + "integrity": "sha512-T0+barUSQRTUQASh8bx02dl+DhF54GtIDY13Y3m9oWTklKbb3Wv974meRpeZ3lp1JpLVECWWNHC4vaG2XHXouQ==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/content-disposition": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-1.1.0.tgz", + "integrity": "sha512-5jRCH9Z/+DRP7rkvY83B+yGIGX96OYdJmzngqnw2SBSxqCFPd0w2km3s5iawpGX8krnwSGmF0FW5Nhr0Hfai3g==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/content-type": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz", + "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie": { + "version": "0.7.2", + "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz", + "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie-signature": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz", + "integrity": "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg==", + "license": "MIT", + "engines": { + "node": ">=6.6.0" + } + }, + "node_modules/cookiejar": { + "version": "2.1.4", + "resolved": "https://registry.npmjs.org/cookiejar/-/cookiejar-2.1.4.tgz", + "integrity": "sha512-LDx6oHrK+PhzLKJU9j5S7/Y3jM/mUHvD/DeI1WQmJn652iPC5Y4TBzC9l+5OMOXlyTTA+SmVUPm0HQUwpD5Jqw==", + "dev": true, + "license": "MIT" + }, + "node_modules/crelt": { + "version": "1.0.7", + "resolved": "https://registry.npmjs.org/crelt/-/crelt-1.0.7.tgz", + "integrity": "sha512-aK6BbWfhf4U/wCcLHKPJl/xa6VkVstRaPywWtMKGwuOLc/wZTyQYuoxgvZnNsBvv7Kg3YTBQYYBCggcviQczuA==", + "dev": true, + "license": "MIT" + }, + "node_modules/css-tree": { + "version": "3.2.1", + "resolved": "https://registry.npmjs.org/css-tree/-/css-tree-3.2.1.tgz", + "integrity": "sha512-X7sjQzceUhu1u7Y/ylrRZFU2FS6LRiFVp6rKLPg23y3x3c3DOKAwuXGDp+PAGjh6CSnCjYeAul8pcT8bAl+lSA==", + "dev": true, + "license": "MIT", + "dependencies": { + "mdn-data": "2.27.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12.20.0 || ^14.13.0 || >=15.0.0" + } + }, + "node_modules/data-urls": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/data-urls/-/data-urls-7.0.0.tgz", + "integrity": "sha512-23XHcCF+coGYevirZceTVD7NdJOqVn+49IHyxgszm+JIiHLoB2TkmPtsYkNWT1pvRSGkc35L6NHs0yHkN2SumA==", + "dev": true, + "license": "MIT", + "dependencies": { + "whatwg-mimetype": "^5.0.0", + "whatwg-url": "^16.0.0" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/data-urls/node_modules/tr46": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/tr46/-/tr46-6.0.0.tgz", + "integrity": "sha512-bLVMLPtstlZ4iMQHpFHTR7GAGj2jxi8Dg0s2h2MafAE4uSWF98FC/3MomU51iQAMf8/qDUbKWf5GxuvvVcXEhw==", + "dev": true, + "license": "MIT", + "dependencies": { + "punycode": "^2.3.1" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/data-urls/node_modules/webidl-conversions": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-8.0.1.tgz", + "integrity": "sha512-BMhLD/Sw+GbJC21C/UgyaZX41nPt8bUTg+jWyDeg7e7YN4xOM05YPSIXceACnXVtqyEw/LMClUQMtMZ+PGGpqQ==", + "dev": true, + "license": "BSD-2-Clause", + "engines": { + "node": ">=20" + } + }, + "node_modules/data-urls/node_modules/whatwg-url": { + "version": "16.0.1", + "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-16.0.1.tgz", + "integrity": "sha512-1to4zXBxmXHV3IiSSEInrreIlu02vUOvrhxJJH5vcxYTBDAx51cqZiKdyTxlecdKNSjj8EcxGBxNf6Vg+945gw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@exodus/bytes": "^1.11.0", + "tr46": "^6.0.0", + "webidl-conversions": "^8.0.1" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/decimal.js": { + "version": "10.6.0", + "resolved": "https://registry.npmjs.org/decimal.js/-/decimal.js-10.6.0.tgz", + "integrity": "sha512-YpgQiITW3JXGntzdUmyUR1V812Hn8T1YVXhCu+wO3OpS4eU9l4YdD3qjyiKdV6mvV29zapkMeD390UVEf2lkUg==", + "dev": true, + "license": "MIT" + }, + "node_modules/delayed-stream": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz", + "integrity": "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.4.0" + } + }, + "node_modules/depd": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", + "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/detect-libc": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", + "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", + "dev": true, + "license": "Apache-2.0", + "peer": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/devtools-protocol": { + "version": "0.0.1687809", + "resolved": "https://registry.npmjs.org/devtools-protocol/-/devtools-protocol-0.0.1687809.tgz", + "integrity": "sha512-t7kSb+UKYdyxcmqy98U2P86ne6rCve3DIJ4+lp54i/hhClgFuG35JrLj23rO14Jw3sWDsp7i+22QhNU0+/qk8g==", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/dezalgo": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/dezalgo/-/dezalgo-1.0.4.tgz", + "integrity": "sha512-rXSP0bf+5n0Qonsb+SVVfNfIsimO4HEtmnIpPHY8Q1UCzKlQrDMfdobr8nJOOsRgWCyMRqeSBQzmWUMq7zvVig==", + "dev": true, + "license": "ISC", + "dependencies": { + "asap": "^2.0.0", + "wrappy": "1" + } + }, + "node_modules/dunder-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", + "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.1", + "es-errors": "^1.3.0", + "gopd": "^1.2.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/ee-first": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", + "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==", + "license": "MIT" + }, + "node_modules/emoji-regex": { + "version": "10.6.0", + "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-10.6.0.tgz", + "integrity": "sha512-toUI84YS5YmxW219erniWD0CIVOo46xGKColeNQRgOzDorgBi1v4D71/OFzgD9GO2UGKIv1C3Sp8DAn0+j5w7A==", + "dev": true, + "license": "MIT" + }, + "node_modules/encodeurl": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz", + "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/entities": { + "version": "8.1.0", + "resolved": "https://registry.npmjs.org/entities/-/entities-8.1.0.tgz", + "integrity": "sha512-kxL7msIffSuh9aaFAMD7rxAIuTRMAHMeBtgHW2yUdWw732ZNh4MehkF2gdjvtdmikkaIP9bFDDJOPlsvm7avrA==", + "dev": true, + "license": "BSD-2-Clause", + "engines": { + "node": ">=20.19.0" + }, + "funding": { + "url": "https://github.com/fb55/entities?sponsor=1" + } + }, + "node_modules/es-define-property": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", + "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-errors": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", + "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-module-lexer": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-2.3.2.tgz", + "integrity": "sha512-poHGpORABojJJucnV9KbOavETW8lBVnphkW77ER5/BQ5Fz7oXSoCNek7IH3vR5nRjdsEz926ibFYX8KtLQmdyw==", + "dev": true, + "license": "MIT" + }, + "node_modules/es-object-atoms": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.2.tgz", + "integrity": "sha512-HWcBoN6NileqtSydK2FqHbS/LoDd2pqrnQHLyJzBj4kOp/ky2MWMN694xOfkK8/SnUsW2DH7EfyVlydKCsm1Zw==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-set-tostringtag": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz", + "integrity": "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==", + "dev": true, + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.6", + "has-tostringtag": "^1.0.2", + "hasown": "^2.0.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/esbuild": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.28.2.tgz", + "integrity": "sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.28.2", + "@esbuild/android-arm": "0.28.2", + "@esbuild/android-arm64": "0.28.2", + "@esbuild/android-x64": "0.28.2", + "@esbuild/darwin-arm64": "0.28.2", + "@esbuild/darwin-x64": "0.28.2", + "@esbuild/freebsd-arm64": "0.28.2", + "@esbuild/freebsd-x64": "0.28.2", + "@esbuild/linux-arm": "0.28.2", + "@esbuild/linux-arm64": "0.28.2", + "@esbuild/linux-ia32": "0.28.2", + "@esbuild/linux-loong64": "0.28.2", + "@esbuild/linux-mips64el": "0.28.2", + "@esbuild/linux-ppc64": "0.28.2", + "@esbuild/linux-riscv64": "0.28.2", + "@esbuild/linux-s390x": "0.28.2", + "@esbuild/linux-x64": "0.28.2", + "@esbuild/netbsd-arm64": "0.28.2", + "@esbuild/netbsd-x64": "0.28.2", + "@esbuild/openbsd-arm64": "0.28.2", + "@esbuild/openbsd-x64": "0.28.2", + "@esbuild/openharmony-arm64": "0.28.2", + "@esbuild/sunos-x64": "0.28.2", + "@esbuild/win32-arm64": "0.28.2", + "@esbuild/win32-ia32": "0.28.2", + "@esbuild/win32-x64": "0.28.2" + } + }, + "node_modules/escalade": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", + "integrity": "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/escape-html": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz", + "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==", + "license": "MIT" + }, + "node_modules/estree-walker": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz", + "integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0" + } + }, + "node_modules/etag": { + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", + "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/expect-type": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz", + "integrity": "sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=12.0.0" + } + }, + "node_modules/express": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz", + "integrity": "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==", + "license": "MIT", + "dependencies": { + "accepts": "^2.0.0", + "body-parser": "^2.2.1", + "content-disposition": "^1.0.0", + "content-type": "^1.0.5", + "cookie": "^0.7.1", + "cookie-signature": "^1.2.1", + "debug": "^4.4.0", + "depd": "^2.0.0", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "etag": "^1.8.1", + "finalhandler": "^2.1.0", + "fresh": "^2.0.0", + "http-errors": "^2.0.0", + "merge-descriptors": "^2.0.0", + "mime-types": "^3.0.0", + "on-finished": "^2.4.1", + "once": "^1.4.0", + "parseurl": "^1.3.3", + "proxy-addr": "^2.0.7", + "qs": "^6.14.0", + "range-parser": "^1.2.1", + "router": "^2.2.0", + "send": "^1.1.0", + "serve-static": "^2.2.0", + "statuses": "^2.0.1", + "type-is": "^2.0.1", + "vary": "^1.1.2" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/fast-safe-stringify": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/fast-safe-stringify/-/fast-safe-stringify-2.1.1.tgz", + "integrity": "sha512-W+KJc2dmILlPplD/H4K9l9LcAHAfPtP6BY84uVLXQ6Evcz9Lcg33Y2z1IVblT6xdY54PXYVHEv+0Wpq8Io6zkA==", + "dev": true, + "license": "MIT" + }, + "node_modules/fdir": { + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", + "integrity": "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12.0.0" + }, + "peerDependencies": { + "picomatch": "^3 || ^4" + }, + "peerDependenciesMeta": { + "picomatch": { + "optional": true + } + } + }, + "node_modules/finalhandler": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-2.1.1.tgz", + "integrity": "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.0", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "on-finished": "^2.4.1", + "parseurl": "^1.3.3", + "statuses": "^2.0.1" + }, + "engines": { + "node": ">= 18.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/form-data": { + "version": "4.0.6", + "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.6.tgz", + "integrity": "sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "asynckit": "^0.4.0", + "combined-stream": "^1.0.8", + "es-set-tostringtag": "^2.1.0", + "hasown": "^2.0.4", + "mime-types": "^2.1.35" + }, + "engines": { + "node": ">= 6" + } + }, + "node_modules/form-data/node_modules/mime-db": { + "version": "1.52.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz", + "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/form-data/node_modules/mime-types": { + "version": "2.1.35", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz", + "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==", + "dev": true, + "license": "MIT", + "dependencies": { + "mime-db": "1.52.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/formidable": { + "version": "3.5.4", + "resolved": "https://registry.npmjs.org/formidable/-/formidable-3.5.4.tgz", + "integrity": "sha512-YikH+7CUTOtP44ZTnUhR7Ic2UASBPOqmaRkRKxRbywPTe5VxF7RRCck4af9wutiZ/QKM5nME9Bie2fFaPz5Gug==", + "dev": true, + "license": "MIT", + "dependencies": { + "@paralleldrive/cuid2": "^2.2.2", + "dezalgo": "^1.0.4", + "once": "^1.4.0" + }, + "engines": { + "node": ">=14.0.0" + }, + "funding": { + "url": "https://ko-fi.com/tunnckoCore/commissions" + } + }, + "node_modules/forwarded": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", + "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/fresh": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/fresh/-/fresh-2.0.0.tgz", + "integrity": "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "peer": true, + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/function-bind": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", + "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-caller-file": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz", + "integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==", + "dev": true, + "license": "ISC", + "engines": { + "node": "6.* || 8.* || >= 10.*" + } + }, + "node_modules/get-east-asian-width": { + "version": "1.7.0", + "resolved": "https://registry.npmjs.org/get-east-asian-width/-/get-east-asian-width-1.7.0.tgz", + "integrity": "sha512-XjH1AECxf0giL2V1aU8vKyRR2ppRUb5c0EvT7zuJTokQ74bNo52zOtghqdWIqrhUD79fo3x0WfKZdOqxF6LG1Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/get-intrinsic": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", + "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "es-define-property": "^1.0.1", + "es-errors": "^1.3.0", + "es-object-atoms": "^1.1.1", + "function-bind": "^1.1.2", + "get-proto": "^1.0.1", + "gopd": "^1.2.0", + "has-symbols": "^1.1.0", + "hasown": "^2.0.2", + "math-intrinsics": "^1.1.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", + "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==", + "license": "MIT", + "dependencies": { + "dunder-proto": "^1.0.1", + "es-object-atoms": "^1.0.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/gopd": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", + "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/has-symbols": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", + "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/has-tostringtag": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/has-tostringtag/-/has-tostringtag-1.0.2.tgz", + "integrity": "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==", + "dev": true, + "license": "MIT", + "dependencies": { + "has-symbols": "^1.0.3" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/hasown": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz", + "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==", + "license": "MIT", + "dependencies": { + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/html-encoding-sniffer": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-6.0.0.tgz", + "integrity": "sha512-CV9TW3Y3f8/wT0BRFc1/KAVQ3TUHiXmaAb6VW9vtiMFf7SLoMd1PdAc4W3KFOFETBJUb90KatHqlsZMWV+R9Gg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@exodus/bytes": "^1.6.0" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/http-errors": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", + "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==", + "license": "MIT", + "dependencies": { + "depd": "~2.0.0", + "inherits": "~2.0.4", + "setprototypeof": "~1.2.0", + "statuses": "~2.0.2", + "toidentifier": "~1.0.1" + }, + "engines": { + "node": ">= 0.8" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/iconv-lite": { + "version": "0.7.3", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.3.tgz", + "integrity": "sha512-IKXpvIzjnC9XTAUbVBcMfGS0EPaIXtW6v+zr+RRp+hqULEpo0owZax6wyRwPOJbWbzjYspQwusTsfVr0ifh4uQ==", + "license": "MIT", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3.0.0" + }, + "engines": { + "node": ">=0.10.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/inherits": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", + "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", + "license": "ISC" + }, + "node_modules/ipaddr.js": { + "version": "1.9.1", + "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", + "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==", + "license": "MIT", + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/is-potential-custom-element-name": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/is-potential-custom-element-name/-/is-potential-custom-element-name-1.0.1.tgz", + "integrity": "sha512-bCYeRA2rVibKZd+s2625gGnGF/t7DSqDs4dP7CrLA1m7jKWz6pps0LpYLJN8Q64HtmPKJ1hrN3nzPNKFEKOUiQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/is-promise": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/is-promise/-/is-promise-4.0.0.tgz", + "integrity": "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==", + "license": "MIT" + }, + "node_modules/jsdom": { + "version": "29.1.1", + "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-29.1.1.tgz", + "integrity": "sha512-ECi4Fi2f7BdJtUKTflYRTiaMxIB0O6zfR1fX0GXpUrf6flp8QIYn1UT20YQqdSOfk2dfkCwS8LAFoJDEppNK5Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "@asamuzakjp/css-color": "^5.1.11", + "@asamuzakjp/dom-selector": "^7.1.1", + "@bramus/specificity": "^2.4.2", + "@csstools/css-syntax-patches-for-csstree": "^1.1.3", + "@exodus/bytes": "^1.15.0", + "css-tree": "^3.2.1", + "data-urls": "^7.0.0", + "decimal.js": "^10.6.0", + "html-encoding-sniffer": "^6.0.0", + "is-potential-custom-element-name": "^1.0.1", + "lru-cache": "^11.3.5", + "parse5": "^8.0.1", + "saxes": "^6.0.0", + "symbol-tree": "^3.2.4", + "tough-cookie": "^6.0.1", + "undici": "^7.25.0", + "w3c-xmlserializer": "^5.0.0", + "webidl-conversions": "^8.0.1", + "whatwg-mimetype": "^5.0.0", + "whatwg-url": "^16.0.1", + "xml-name-validator": "^5.0.0" + }, + "engines": { + "node": "^20.19.0 || ^22.13.0 || >=24.0.0" + }, + "peerDependencies": { + "canvas": "^3.0.0" + }, + "peerDependenciesMeta": { + "canvas": { + "optional": true + } + } + }, + "node_modules/jsdom/node_modules/tr46": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/tr46/-/tr46-6.0.0.tgz", + "integrity": "sha512-bLVMLPtstlZ4iMQHpFHTR7GAGj2jxi8Dg0s2h2MafAE4uSWF98FC/3MomU51iQAMf8/qDUbKWf5GxuvvVcXEhw==", + "dev": true, + "license": "MIT", + "dependencies": { + "punycode": "^2.3.1" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/jsdom/node_modules/webidl-conversions": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-8.0.1.tgz", + "integrity": "sha512-BMhLD/Sw+GbJC21C/UgyaZX41nPt8bUTg+jWyDeg7e7YN4xOM05YPSIXceACnXVtqyEw/LMClUQMtMZ+PGGpqQ==", + "dev": true, + "license": "BSD-2-Clause", + "engines": { + "node": ">=20" + } + }, + "node_modules/jsdom/node_modules/whatwg-url": { + "version": "16.0.1", + "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-16.0.1.tgz", + "integrity": "sha512-1to4zXBxmXHV3IiSSEInrreIlu02vUOvrhxJJH5vcxYTBDAx51cqZiKdyTxlecdKNSjj8EcxGBxNf6Vg+945gw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@exodus/bytes": "^1.11.0", + "tr46": "^6.0.0", + "webidl-conversions": "^8.0.1" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/lightningcss": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.33.0.tgz", + "integrity": "sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA==", + "dev": true, + "license": "MPL-2.0", + "peer": true, + "dependencies": { + "detect-libc": "^2.0.3" + }, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + }, + "optionalDependencies": { + "lightningcss-android-arm64": "1.33.0", + "lightningcss-darwin-arm64": "1.33.0", + "lightningcss-darwin-x64": "1.33.0", + "lightningcss-freebsd-x64": "1.33.0", + "lightningcss-linux-arm-gnueabihf": "1.33.0", + "lightningcss-linux-arm64-gnu": "1.33.0", + "lightningcss-linux-arm64-musl": "1.33.0", + "lightningcss-linux-x64-gnu": "1.33.0", + "lightningcss-linux-x64-musl": "1.33.0", + "lightningcss-win32-arm64-msvc": "1.33.0", + "lightningcss-win32-x64-msvc": "1.33.0" + } + }, + "node_modules/lightningcss-android-arm64": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-android-arm64/-/lightningcss-android-arm64-1.33.0.tgz", + "integrity": "sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "android" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-darwin-arm64": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.33.0.tgz", + "integrity": "sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "darwin" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-darwin-x64": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.33.0.tgz", + "integrity": "sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "darwin" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-freebsd-x64": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.33.0.tgz", + "integrity": "sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-arm-gnueabihf": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.33.0.tgz", + "integrity": "sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-arm64-gnu": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.33.0.tgz", + "integrity": "sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-arm64-musl": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.33.0.tgz", + "integrity": "sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-x64-gnu": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.33.0.tgz", + "integrity": "sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-x64-musl": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.33.0.tgz", + "integrity": "sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-win32-arm64-msvc": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.33.0.tgz", + "integrity": "sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "win32" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-win32-x64-msvc": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.33.0.tgz", + "integrity": "sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "win32" + ], + "peer": true, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lru-cache": { + "version": "11.5.3", + "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.5.3.tgz", + "integrity": "sha512-U4N8FgzmWxc8k1VH8Kr6lQg18U7Fjvby6wXHVRX/ZZ7IwWbRMgrRbP0Wrb5q5NVinryp4SQampHKdvtecItxUg==", + "dev": true, + "license": "BlueOak-1.0.0", + "engines": { + "node": "20 || >=22" + } + }, + "node_modules/magic-string": { + "version": "1.4.2", + "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-1.4.2.tgz", + "integrity": "sha512-vG+rjFRj1PqdIBozIxAGMjPlOhaVe+GXpbttY/iSK7rGcJRMlwNJO7dcUwmUqkymsFLJiNGI06t4D7Fr7yRC9g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.6.0" + } + }, + "node_modules/math-intrinsics": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", + "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/mdn-data": { + "version": "2.27.1", + "resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.27.1.tgz", + "integrity": "sha512-9Yubnt3e8A0OKwxYSXyhLymGW4sCufcLG6VdiDdUGVkPhpqLxlvP5vl1983gQjJl3tqbrM731mjaZaP68AgosQ==", + "dev": true, + "license": "CC0-1.0" + }, + "node_modules/media-typer": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-1.1.1.tgz", + "integrity": "sha512-yz3xRaG20c6/BOzvYoDaGtPmGscs7YivItZEEqe6GbwNfHuxu9YNmvnEkMzKldAGY4/80pRcQRZSEnhquk9XuQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/memory-pager": { + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/memory-pager/-/memory-pager-1.5.0.tgz", + "integrity": "sha512-ZS4Bp4r/Zoeq6+NLJpP+0Zzm0pR8whtGPf1XExKLJBAczGMnSi3It14OiNCStjQjM6NU1okjQGSxgEZN8eBYKg==", + "license": "MIT" + }, + "node_modules/merge-descriptors": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-2.0.0.tgz", + "integrity": "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/methods": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz", + "integrity": "sha512-iclAHeNqNm68zFtnZ0e+1L2yUIdvzNoauKU4WBA3VvH/vPFieF7qfRlwUZU+DA9P9bPXIS90ulxoUoCH23sV2w==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/mime": { + "version": "2.6.0", + "resolved": "https://registry.npmjs.org/mime/-/mime-2.6.0.tgz", + "integrity": "sha512-USPkMeET31rOMiarsBNIHZKLGgvKc/LrjofAnBlOttf5ajRvqiRA8QsenbcooctK6d6Ts6aqZXBA+XbkKthiQg==", + "dev": true, + "license": "MIT", + "bin": { + "mime": "cli.js" + }, + "engines": { + "node": ">=4.0.0" + } + }, + "node_modules/mime-db": { + "version": "1.54.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz", + "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/mime-types": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz", + "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==", + "license": "MIT", + "dependencies": { + "mime-db": "^1.54.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/mitt": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/mitt/-/mitt-3.0.1.tgz", + "integrity": "sha512-vKivATfr97l2/QBCYAkXYDbrIWPM2IIKEl7YPhjCvKlG3kE2gm+uBo6nEXK3M5/Ffh/FLpKExzOQ3JJoJGFKBw==", + "dev": true, + "license": "MIT" + }, + "node_modules/modern-tar": { + "version": "0.8.5", + "resolved": "https://registry.npmjs.org/modern-tar/-/modern-tar-0.8.5.tgz", + "integrity": "sha512-snEhs+6G5Tjd4I7tLCDOaoln2RgE0bD19RzEKgvgK2hZ5VKy3MpLhLTZ2fWpXSTg4K2cyPwp+VHATFJhxfnOeA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/mongodb": { + "version": "7.7.0", + "resolved": "https://registry.npmjs.org/mongodb/-/mongodb-7.7.0.tgz", + "integrity": "sha512-XG1xXEqyouAV8GCb3PuRaujofeXKpAJAPvXYWZgHl6Dz4QHSJ1nRLl0+GpqaLTwKvC0RFE9truOur34RNFY1nw==", + "license": "Apache-2.0", + "dependencies": { + "@mongodb-js/saslprep": "^1.5.0", + "bson": "^7.3.2", + "mongodb-connection-string-url": "^7.0.2" + }, + "engines": { + "node": ">=20.19.0" + }, + "peerDependencies": { + "@aws-sdk/credential-providers": "^3.806.0", + "@mongodb-js/zstd": "^7.0.0", + "gcp-metadata": "^7.0.1", + "kerberos": "^7.0.0", + "mongodb-client-encryption": "^7.2.0", + "snappy": "^7.3.2", + "socks": "^2.8.6" + }, + "peerDependenciesMeta": { + "@aws-sdk/credential-providers": { + "optional": true + }, + "@mongodb-js/zstd": { + "optional": true + }, + "gcp-metadata": { + "optional": true + }, + "kerberos": { + "optional": true + }, + "mongodb-client-encryption": { + "optional": true + }, + "snappy": { + "optional": true + }, + "socks": { + "optional": true + } + } + }, + "node_modules/mongodb-connection-string-url": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/mongodb-connection-string-url/-/mongodb-connection-string-url-7.0.2.tgz", + "integrity": "sha512-ZoS07RoFqpKYQwAk59qmrx8+jJHNHU30UjlU96QktiGn1ltvDr+vCznLX5DiUBLEpMAHatHNWV1nM/74ul66kA==", + "license": "Apache-2.0", + "dependencies": { + "@types/whatwg-url": "^13.0.0", + "whatwg-url": "^14.1.0" + }, + "engines": { + "node": ">=20.19.0" + } + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, + "node_modules/nanoid": { + "version": "3.3.19", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.19.tgz", + "integrity": "sha512-Y2tUNy4ouw6tq5oDSKeQYGOyhkUBhNOcGV/02KC+6kd9eDGqdZd++mjMiIDilrBYvjEnCYvVtsuHCuP+okSfug==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "peer": true, + "bin": { + "nanoid": "bin/nanoid.cjs" + }, + "engines": { + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" + } + }, + "node_modules/negotiator": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.1.0.tgz", + "integrity": "sha512-NMPBRMJgiQHjbd8phG3Vebdx4kZ1H121rbl5IkMqeOsahptB9BKo/d7oJ3zTXqTgagn2bWlNSXkh0QUGM31RYg==", + "license": "MIT", + "dependencies": { + "content-type": "^2.1.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/negotiator/node_modules/content-type": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz", + "integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/object-inspect": { + "version": "1.13.4", + "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", + "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/obug": { + "version": "2.2.1", + "resolved": "https://registry.npmjs.org/obug/-/obug-2.2.1.tgz", + "integrity": "sha512-XrsrhT5sybtKI6wakr2SPOlGZWWYbUXZ7a0jT8/QOeAPau+1X/bSegNe5YR75oJmEZQbKningirmGOEJCIk61Q==", + "dev": true, + "funding": [ + "https://github.com/sponsors/sxzz", + "https://opencollective.com/debug" + ], + "license": "MIT", + "engines": { + "node": ">=12.20.0" + } + }, + "node_modules/on-finished": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", + "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==", + "license": "MIT", + "dependencies": { + "ee-first": "1.1.1" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/once": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", + "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", + "license": "ISC", + "dependencies": { + "wrappy": "1" + } + }, + "node_modules/oracledb": { + "version": "7.0.1", + "resolved": "https://registry.npmjs.org/oracledb/-/oracledb-7.0.1.tgz", + "integrity": "sha512-xlM0Ceh6A5stQLAdEfKf3pgCSkbOjQLo2ZPEi3+kXklz+KbZD3fLi/nsTSbQeZZNFBSFDNxdn1Ek3+bxG40M8w==", + "hasInstallScript": true, + "license": "(Apache-2.0 OR UPL-1.0)", + "engines": { + "node": ">=14.17" + } + }, + "node_modules/parse5": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/parse5/-/parse5-8.0.1.tgz", + "integrity": "sha512-z1e/HMG90obSGeidlli3hj7cbocou0/wa5HacvI3ASx34PecNjNQeaHNo5WIZpWofN9kgkqV1q5YvXe3F0FoPw==", + "dev": true, + "license": "MIT", + "dependencies": { + "entities": "^8.0.0" + }, + "funding": { + "url": "https://github.com/inikulin/parse5?sponsor=1" + } + }, + "node_modules/parseurl": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", + "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/path-to-regexp": { + "version": "8.4.2", + "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-8.4.2.tgz", + "integrity": "sha512-qRcuIdP69NPm4qbACK+aDogI5CBDMi1jKe0ry5rSQJz8JVLsC7jV8XpiJjGRLLol3N+R5ihGYcrPLTno6pAdBA==", + "license": "MIT", + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC", + "peer": true + }, + "node_modules/picomatch": { + "version": "4.0.7", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.7.tgz", + "integrity": "sha512-qcJu88Q2IWqJsDD529JKMdwGm/dvInW4HvQnRwiH9JtihJvzGOscDtHE3x1pBKeUOTysQ8kVmLnJ2kJu7yhcGA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/postcss": { + "version": "8.5.28", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.28.tgz", + "integrity": "sha512-RRuzqDtt5Y9h3quz5hWhK+TPnsmVs6WwSU6LkJMeY4HstUEDuYTG8UJSdawMRzmzAtV+KEoG8N3Qg2qLy5vM/A==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "peer": true, + "dependencies": { + "nanoid": "^3.3.18", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/proxy-addr": { + "version": "2.0.8", + "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz", + "integrity": "sha512-5nnx0yGyVUcY6t9RnWcARWtwT9F1D8O9rt08htPvnd49W1IgZtmLkhu9WfMzQj1cFxjHIO6connUNVW5k7AVyQ==", + "license": "MIT", + "dependencies": { + "forwarded": "0.2.0", + "ipaddr.js": "1.9.1" + }, + "engines": { + "node": ">= 0.10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/punycode": { + "version": "2.3.1", + "resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz", + "integrity": "sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg==", + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/puppeteer-core": { + "version": "25.12.0", + "resolved": "https://registry.npmjs.org/puppeteer-core/-/puppeteer-core-25.12.0.tgz", + "integrity": "sha512-z6LQUt5SH7jwGtGA+nMtJuEjZIsqPiwQ+EFHCYuVDQrJpvifNmmGuyDvTzAKPyFzihxgHc5f3aMzu2kn8Ps37A==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@puppeteer/browsers": "3.2.3", + "chromium-bidi": "17.0.2", + "devtools-protocol": "0.0.1687809", + "typed-query-selector": "^2.12.2", + "webdriver-bidi-protocol": "0.4.3", + "ws": "^8.21.3" + }, + "engines": { + "node": ">=22.12.0" + } + }, + "node_modules/qs": { + "version": "6.16.0", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz", + "integrity": "sha512-h6fhOIaRrID2CbEY2fqs+7t+UXZo+MLAnU5gRIq85uFtdiUPCdsApMlHhXogKVM4HM2DVbIjGNTTYH2OcmP1vA==", + "license": "BSD-3-Clause", + "dependencies": { + "es-define-property": "^1.0.1", + "side-channel": "^1.1.1" + }, + "engines": { + "node": ">=0.6" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/range-parser": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.3.0.tgz", + "integrity": "sha512-hek2mFQpPuI4E1BBKrSto+BU3e3x4xuarsbiwr3+lf7p44juvFMV0XFWQAP3xUyqXA4RrXLIoaSUGbSt056ZMw==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/raw-body": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-3.0.2.tgz", + "integrity": "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==", + "license": "MIT", + "dependencies": { + "bytes": "~3.1.2", + "http-errors": "~2.0.1", + "iconv-lite": "~0.7.0", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/require-from-string": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", + "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/rolldown": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.2.11.tgz", + "integrity": "sha512-qpSwIyz0jHQq5qXBTNxFmE6664rJ7O+4TvPFOiOaBSrz8IOHc1koKKSqTM2H6u1UG1+TveuC6vaDHKXFOvb1Kw==", + "dev": true, + "license": "MIT", + "peer": true, + "dependencies": { + "@oxc-project/types": "=0.151.0", + "@rolldown/pluginutils": "^1.0.0" + }, + "bin": { + "rolldown": "bin/cli.mjs" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + }, + "optionalDependencies": { + "@rolldown/binding-android-arm-eabi": "1.2.11", + "@rolldown/binding-android-arm64": "1.2.11", + "@rolldown/binding-darwin-arm64": "1.2.11", + "@rolldown/binding-darwin-x64": "1.2.11", + "@rolldown/binding-freebsd-x64": "1.2.11", + "@rolldown/binding-linux-arm-gnueabihf": "1.2.11", + "@rolldown/binding-linux-arm64-gnu": "1.2.11", + "@rolldown/binding-linux-arm64-musl": "1.2.11", + "@rolldown/binding-linux-ppc64-gnu": "1.2.11", + "@rolldown/binding-linux-s390x-gnu": "1.2.11", + "@rolldown/binding-linux-x64-gnu": "1.2.11", + "@rolldown/binding-linux-x64-musl": "1.2.11", + "@rolldown/binding-openharmony-arm64": "1.2.11", + "@rolldown/binding-win32-arm64-msvc": "1.2.11", + "@rolldown/binding-win32-x64-msvc": "1.2.11" + } + }, + "node_modules/router": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz", + "integrity": "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.0", + "depd": "^2.0.0", + "is-promise": "^4.0.0", + "parseurl": "^1.3.3", + "path-to-regexp": "^8.0.0" + }, + "engines": { + "node": ">= 18" + } + }, + "node_modules/safer-buffer": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", + "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", + "license": "MIT" + }, + "node_modules/saxes": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/saxes/-/saxes-6.0.0.tgz", + "integrity": "sha512-xAg7SOnEhrm5zI3puOOKyy1OMcMlIJZYNJY7xLBwSze0UjhPLnWfj2GF2EpT0jmzaJKIWKHLsaSSajf35bcYnA==", + "dev": true, + "license": "ISC", + "dependencies": { + "xmlchars": "^2.2.0" + }, + "engines": { + "node": ">=v12.22.7" + } + }, + "node_modules/send": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz", + "integrity": "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.3", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "etag": "^1.8.1", + "fresh": "^2.0.0", + "http-errors": "^2.0.1", + "mime-types": "^3.0.2", + "ms": "^2.1.3", + "on-finished": "^2.4.1", + "range-parser": "^1.2.1", + "statuses": "^2.0.2" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/serve-static": { + "version": "2.2.1", + "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-2.2.1.tgz", + "integrity": "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw==", + "license": "MIT", + "dependencies": { + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "parseurl": "^1.3.3", + "send": "^1.2.0" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/setprototypeof": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", + "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==", + "license": "ISC" + }, + "node_modules/side-channel": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz", + "integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.4", + "side-channel-list": "^1.0.1", + "side-channel-map": "^1.0.1", + "side-channel-weakmap": "^1.0.2" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-list": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz", + "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.4" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-map": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz", + "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==", + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-weakmap": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz", + "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==", + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3", + "side-channel-map": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/source-map-js": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", + "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/sparse-bitfield": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/sparse-bitfield/-/sparse-bitfield-3.0.3.tgz", + "integrity": "sha512-kvzhi7vqKTfkh0PZU+2D2PIllw2ymqJKujUcyPMd9Y75Nv4nPbGJZXNhxsgdQab2BmlDct1YnfQCguEvHr7VsQ==", + "license": "MIT", + "dependencies": { + "memory-pager": "^1.0.2" + } + }, + "node_modules/statuses": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", + "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/std-env": { + "version": "4.3.0", + "resolved": "https://registry.npmjs.org/std-env/-/std-env-4.3.0.tgz", + "integrity": "sha512-OtU/EgQ1kIm5KwqQpBC6ZEMXrZRui11w8zgfTWp8cdO9B8OaPsbA8bTHO2P+HNo1VlUTGMVBwPhydu6poeXiag==", + "dev": true, + "license": "MIT" + }, + "node_modules/string-width": { + "version": "8.3.0", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-8.3.0.tgz", + "integrity": "sha512-ZbmZM0JCihQN91dWnxoipT2KOEyHqEyfRXUyjuRhW8b/xnqPDoq4gWEVApTVa9db2wN8mmoikgFBbjh71+cGeQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "get-east-asian-width": "^1.5.0", + "strip-ansi": "^7.1.2" + }, + "engines": { + "node": ">=20" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/strip-ansi": { + "version": "7.2.0", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-7.2.0.tgz", + "integrity": "sha512-yDPMNjp4WyfYBkHnjIRLfca1i6KMyGCtsVgoKe/z1+6vukgaENdgGBZt+ZmKPc4gavvEZ5OgHfHdrazhgNyG7w==", + "dev": true, + "license": "MIT", + "dependencies": { + "ansi-regex": "^6.2.2" + }, + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/chalk/strip-ansi?sponsor=1" + } + }, + "node_modules/style-mod": { + "version": "4.1.4", + "resolved": "https://registry.npmjs.org/style-mod/-/style-mod-4.1.4.tgz", + "integrity": "sha512-XXWIQt633/EpAFx8aZDOTjBzrCaGmhvEQlQo6MVPfa2OzO2cWo+4hV9h+6UkHYlXGfy+ODXKUdP7Pthmcu5ATw==", + "dev": true, + "license": "MIT" + }, + "node_modules/superagent": { + "version": "10.4.1", + "resolved": "https://registry.npmjs.org/superagent/-/superagent-10.4.1.tgz", + "integrity": "sha512-PVMkMrhKrSTtFhUU8jiWuGh/gyRMKiyCTCmm0+qVzRNxlfntmRKJkDhlEXN62x3HIb33sQ4kmL8lcSdccWAchQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "component-emitter": "^1.3.1", + "cookiejar": "^2.1.4", + "debug": "^4.3.7", + "fast-safe-stringify": "^2.1.1", + "form-data": "^4.0.5", + "formidable": "^3.5.4", + "methods": "^1.1.2", + "mime": "2.6.0", + "qs": "^6.14.1" + }, + "engines": { + "node": ">=14.18.0" + } + }, + "node_modules/supertest": { + "version": "7.3.0", + "resolved": "https://registry.npmjs.org/supertest/-/supertest-7.3.0.tgz", + "integrity": "sha512-UwwmWq3xLhyU96c521wYNaBg7IqX78Wpj4FKs6Katp6vdklX42zU5ESvX3KeZRLFyO44RwFp4yvxYMNZzCcX9g==", + "dev": true, + "license": "MIT", + "dependencies": { + "cookie-signature": "^1.2.2", + "methods": "^1.1.2", + "superagent": "^10.3.0" + }, + "engines": { + "node": ">=14.18.0" + } + }, + "node_modules/symbol-tree": { + "version": "3.2.4", + "resolved": "https://registry.npmjs.org/symbol-tree/-/symbol-tree-3.2.4.tgz", + "integrity": "sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinybench": { + "version": "6.2.0", + "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-6.2.0.tgz", + "integrity": "sha512-78U2TlB2CnVenajOFzf3BKSm0J6oz5L0NV7g32LCPccvYc0lbWvys4d3uUUCS2B1N8PAf2+aekR8i1KbC3HO7Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/tinyexec": { + "version": "1.3.1", + "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.3.1.tgz", + "integrity": "sha512-GCvB3aoys96IuDFBMcTB46JOR6mdMtAToqwiW8JlWhsoh1mhHi/xn9ss/Dg7N555GiJyEt2qzoG/NHCwM6h1EA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/tinyglobby": { + "version": "0.2.17", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz", + "integrity": "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==", + "dev": true, + "license": "MIT", + "dependencies": { + "fdir": "^6.5.0", + "picomatch": "^4.0.4" + }, + "engines": { + "node": ">=12.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/SuperchupuDev" + } + }, + "node_modules/tldts": { + "version": "7.4.16", + "resolved": "https://registry.npmjs.org/tldts/-/tldts-7.4.16.tgz", + "integrity": "sha512-QwBER5KMR86IIjpIiO7H/Z3IMJPsZ1A6RKPAqzTTgOyUQUSt9FdnKcqhTaJmkY6HVrgouZHZR0ncK5QxvmnQeg==", + "dev": true, + "license": "MIT", + "dependencies": { + "tldts-core": "^7.4.16" + }, + "bin": { + "tldts": "bin/cli.js" + } + }, + "node_modules/tldts-core": { + "version": "7.4.16", + "resolved": "https://registry.npmjs.org/tldts-core/-/tldts-core-7.4.16.tgz", + "integrity": "sha512-MDolfaSJtlSK5Y0A1xl3277ekubZwobpBjugknDizI9O5Rm60a1m8k4ICK+MRsCDzPygT81mp3BBf5RKDlFRfA==", + "dev": true, + "license": "MIT" + }, + "node_modules/toidentifier": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz", + "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==", + "license": "MIT", + "engines": { + "node": ">=0.6" + } + }, + "node_modules/tough-cookie": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-6.0.2.tgz", + "integrity": "sha512-exgYmnmL/sJpR3upZfXG5PoatXQii55xAiXGXzY+sROLZ/Y+SLcp9PgJNI9Vz37HpQ74WvDcLT8eqm+kV3FzrA==", + "dev": true, + "license": "BSD-3-Clause", + "dependencies": { + "tldts": "^7.0.5" + }, + "engines": { + "node": ">=16" + } + }, + "node_modules/tr46": { + "version": "5.1.1", + "resolved": "https://registry.npmjs.org/tr46/-/tr46-5.1.1.tgz", + "integrity": "sha512-hdF5ZgjTqgAntKkklYw0R03MG2x/bSzTtkxmIRw/sTNV8YXsCJ1tfLAX23lhxhHJlEf3CRCOCGGWw3vI3GaSPw==", + "license": "MIT", + "dependencies": { + "punycode": "^2.3.1" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/type-is": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/type-is/-/type-is-2.1.0.tgz", + "integrity": "sha512-faYHw0anBbc/kWF3zFTEnxSFOAGUX9GFbOBthvDdLsIlEoWOFOtS0zgCiQYwIskL9iGXZL3kAXD8OoZ4GmMATA==", + "license": "MIT", + "dependencies": { + "content-type": "^2.0.0", + "media-typer": "^1.1.0", + "mime-types": "^3.0.0" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/type-is/node_modules/content-type": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz", + "integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/typed-query-selector": { + "version": "2.12.3", + "resolved": "https://registry.npmjs.org/typed-query-selector/-/typed-query-selector-2.12.3.tgz", + "integrity": "sha512-/dtuMZyutHLlrxWhhC1uib00ELUr2dZ8C9B8YzZYsJmeE3V56OxRblrSM2UaF142pcMtak7ehZ6+poWGeehNIg==", + "dev": true, + "license": "MIT" + }, + "node_modules/undici": { + "version": "7.30.0", + "resolved": "https://registry.npmjs.org/undici/-/undici-7.30.0.tgz", + "integrity": "sha512-dkrQXeHSaoamnItlYbmzG0wFYrM0ZwDxCIg0A7aKjTyyhh9svRzCNFEzV+Vm05/yehjCzjDZ31KXfGEjYSztDQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20.18.1" + } + }, + "node_modules/unpipe": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz", + "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/vary": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", + "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/vite": { + "version": "8.3.1", + "resolved": "https://registry.npmjs.org/vite/-/vite-8.3.1.tgz", + "integrity": "sha512-/bvH9E9tmCXRGp2uXY3WbOldqpTwFkbha/8ANaEQ6VkxhH60KyqLwgZq6lG2y+4uT55x9+9eUHMpQ7uGnOCKjA==", + "dev": true, + "license": "MIT", + "peer": true, + "dependencies": { + "lightningcss": "^1.33.0", + "picomatch": "^4.0.7", + "postcss": "^8.5.28", + "rolldown": "~1.2.9", + "tinyglobby": "^0.2.17" + }, + "bin": { + "vite": "bin/vite.js" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + }, + "funding": { + "url": "https://github.com/vitejs/vite?sponsor=1" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + }, + "peerDependencies": { + "@types/node": "^20.19.0 || >=22.12.0", + "@vitejs/devtools": "^0.7.1", + "esbuild": "^0.27.0 || ^0.28.0", + "jiti": ">=1.21.0", + "less": "^4.0.0", + "sass": "^1.70.0", + "sass-embedded": "^1.70.0", + "stylus": ">=0.54.8", + "sugarss": "^5.0.0", + "terser": "^5.16.0", + "tsx": "^4.8.1", + "yaml": "^2.4.2" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "@vitejs/devtools": { + "optional": true + }, + "esbuild": { + "optional": true + }, + "jiti": { + "optional": true + }, + "less": { + "optional": true + }, + "sass": { + "optional": true + }, + "sass-embedded": { + "optional": true + }, + "stylus": { + "optional": true + }, + "sugarss": { + "optional": true + }, + "terser": { + "optional": true + }, + "tsx": { + "optional": true + }, + "yaml": { + "optional": true + } + } + }, + "node_modules/vitest": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/vitest/-/vitest-5.0.2.tgz", + "integrity": "sha512-7MQrx9pDv5aHiUcovIb/70Ys3tgtkUVgCtledvKdCmEO+/1Dicq5ZqoSxOW034m03oqC+oHOKui2dM6qtMLoJg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/chai": "^5.2.2", + "@vitest/mocker": "5.0.2", + "chai": "^6.2.2", + "es-module-lexer": "^2.3.2", + "expect-type": "^1.4.0", + "magic-string": "^1.2.3", + "obug": "^2.1.4", + "picomatch": "^4.0.7", + "std-env": "^4.2.0", + "tinybench": "^6.1.4", + "tinyexec": "^1.3.0", + "tinyglobby": "^0.2.17", + "why-is-node-running": "^3.2.1" + }, + "bin": { + "vitest": "vitest.mjs" + }, + "engines": { + "node": "^22.12.0 || ^24.0.0 || >=26.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "@edge-runtime/vm": "*", + "@opentelemetry/api": "^1.9.0", + "@types/node": "^22.0.0 || >=24.0.0", + "@vitest/browser-playwright": "5.0.2", + "@vitest/browser-preview": "5.0.2", + "@vitest/browser-webdriverio": "^5.0.0-beta.5 || >=5.0.0", + "@vitest/coverage-istanbul": "5.0.2", + "@vitest/coverage-v8": "5.0.2", + "@vitest/ui": "5.0.2", + "happy-dom": "*", + "jsdom": "*", + "vite": "^6.4.0 || ^7.0.0 || ^8.0.0" + }, + "peerDependenciesMeta": { + "@edge-runtime/vm": { + "optional": true + }, + "@opentelemetry/api": { + "optional": true + }, + "@types/node": { + "optional": true + }, + "@vitest/browser-playwright": { + "optional": true + }, + "@vitest/browser-preview": { + "optional": true + }, + "@vitest/browser-webdriverio": { + "optional": true + }, + "@vitest/coverage-istanbul": { + "optional": true + }, + "@vitest/coverage-v8": { + "optional": true + }, + "@vitest/ui": { + "optional": true + }, + "happy-dom": { + "optional": true + }, + "jsdom": { + "optional": true + }, + "vite": { + "optional": false + } + } + }, + "node_modules/w3c-keyname": { + "version": "2.2.8", + "resolved": "https://registry.npmjs.org/w3c-keyname/-/w3c-keyname-2.2.8.tgz", + "integrity": "sha512-dpojBhNsCNN7T82Tm7k26A6G9ML3NkhDsnw9n/eoxSRlVBB4CEtIQ/KTCLI2Fwf3ataSXRhYFkQi3SlnFwPvPQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/w3c-xmlserializer": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/w3c-xmlserializer/-/w3c-xmlserializer-5.0.0.tgz", + "integrity": "sha512-o8qghlI8NZHU1lLPrpi2+Uq7abh4GGPpYANlalzWxyWteJOCsr/P+oPBA49TOLu5FTZO4d3F9MnWJfiMo4BkmA==", + "dev": true, + "license": "MIT", + "dependencies": { + "xml-name-validator": "^5.0.0" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/webdriver-bidi-protocol": { + "version": "0.4.3", + "resolved": "https://registry.npmjs.org/webdriver-bidi-protocol/-/webdriver-bidi-protocol-0.4.3.tgz", + "integrity": "sha512-uuN0goWfxP22B7J/uAgBpOYNPttC+XVseYE+rSY5+rQ+YBeVz/VORw8WbmLVcqW78zNg5A4qnjNXYUWR3il2ig==", + "dev": true, + "license": "Apache-2.0" + }, + "node_modules/webidl-conversions": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-7.0.0.tgz", + "integrity": "sha512-VwddBukDzu71offAQR975unBIGqfKZpM+8ZX6ySk8nYhVoo5CYaZyzt3YBvYtRtO+aoGlqxPg/B87NGVZ/fu6g==", + "license": "BSD-2-Clause", + "engines": { + "node": ">=12" + } + }, + "node_modules/whatwg-mimetype": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-5.0.0.tgz", + "integrity": "sha512-sXcNcHOC51uPGF0P/D4NVtrkjSU2fNsm9iog4ZvZJsL3rjoDAzXZhkm2MWt1y+PUdggKAYVoMAIYcs78wJ51Cw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20" + } + }, + "node_modules/whatwg-url": { + "version": "14.2.0", + "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-14.2.0.tgz", + "integrity": "sha512-De72GdQZzNTUBBChsXueQUnPKDkg/5A5zp7pFDuQAj5UFoENpiACU0wlCvzpAGnTkj++ihpKwKyYewn/XNUbKw==", + "license": "MIT", + "dependencies": { + "tr46": "^5.1.0", + "webidl-conversions": "^7.0.0" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/why-is-node-running": { + "version": "3.2.2", + "resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-3.2.2.tgz", + "integrity": "sha512-NKUzAelcoCXhXL4dJzKIwXeR8iEVqsA0Lq6Vnd0UXvgaKbzVo4ZTHROF2Jidrv+SgxOQ03fMinnNhzZATxOD3A==", + "dev": true, + "license": "MIT", + "bin": { + "why-is-node-running": "cli.js" + }, + "engines": { + "node": ">=20.11" + } + }, + "node_modules/wrap-ansi": { + "version": "9.0.2", + "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-9.0.2.tgz", + "integrity": "sha512-42AtmgqjV+X1VpdOfyTGOYRi0/zsoLqtXQckTmqTeybT+BDIbM/Guxo7x3pE2vtpr1ok6xRqM9OpBe+Jyoqyww==", + "dev": true, + "license": "MIT", + "dependencies": { + "ansi-styles": "^6.2.1", + "string-width": "^7.0.0", + "strip-ansi": "^7.1.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/chalk/wrap-ansi?sponsor=1" + } + }, + "node_modules/wrap-ansi/node_modules/string-width": { + "version": "7.2.0", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-7.2.0.tgz", + "integrity": "sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "emoji-regex": "^10.3.0", + "get-east-asian-width": "^1.0.0", + "strip-ansi": "^7.1.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/wrappy": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", + "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", + "license": "ISC" + }, + "node_modules/ws": { + "version": "8.22.0", + "resolved": "https://registry.npmjs.org/ws/-/ws-8.22.0.tgz", + "integrity": "sha512-Ydggc987+RO0AnWtZ/7Wq9FtNvcrL1b/RO0ud9mWjUPgDrsAAwQSF51sm2hm1XofbU/4jkpGEsLFsZZxU+1DOg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=10.0.0" + }, + "peerDependencies": { + "bufferutil": "^4.0.1", + "utf-8-validate": ">=5.0.2" + }, + "peerDependenciesMeta": { + "bufferutil": { + "optional": true + }, + "utf-8-validate": { + "optional": true + } + } + }, + "node_modules/xml-name-validator": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/xml-name-validator/-/xml-name-validator-5.0.0.tgz", + "integrity": "sha512-EvGK8EJ3DhaHfbRlETOWAS5pO9MZITeauHKJyb8wyajUfQUenkIg2MvLDTZ4T/TgIcm3HU0TFBgWWboAZ30UHg==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=18" + } + }, + "node_modules/xmlchars": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/xmlchars/-/xmlchars-2.2.0.tgz", + "integrity": "sha512-JZnDKK8B0RCDw84FNdDAIpZK+JuJw+s7Lz8nksI7SIuU3UXJJslUthsi+uWBUYOwPFwW7W7PRLRfUKpxjtjFCw==", + "dev": true, + "license": "MIT" + }, + "node_modules/y18n": { + "version": "5.0.8", + "resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.8.tgz", + "integrity": "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==", + "dev": true, + "license": "ISC", + "engines": { + "node": ">=10" + } + }, + "node_modules/yaml": { + "version": "2.9.1", + "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.1.tgz", + "integrity": "sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==", + "license": "ISC", + "bin": { + "yaml": "bin.mjs" + }, + "engines": { + "node": ">= 14.6" + }, + "funding": { + "url": "https://github.com/sponsors/eemeli" + } + }, + "node_modules/yargs": { + "version": "18.2.0", + "resolved": "https://registry.npmjs.org/yargs/-/yargs-18.2.0.tgz", + "integrity": "sha512-9OpKOLeaoNFecEp7P6iYbzze/5CqWoH7N3SMs/6y1XF4nMvFMspEGZzJ6uFi9MmQwXhyzqYoSEKO3tvA3Z/o2w==", + "dev": true, + "license": "MIT", + "dependencies": { + "cliui": "^9.0.1", + "escalade": "^3.1.1", + "get-caller-file": "^2.0.5", + "string-width": "^8.2.1", + "y18n": "^5.0.5", + "yargs-parser": "^22.0.0" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=23" + } + }, + "node_modules/yargs-parser": { + "version": "22.0.0", + "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-22.0.0.tgz", + "integrity": "sha512-rwu/ClNdSMpkSrUb+d6BRsSkLUq1fmfsY6TOpYzTwvwkg1/NRG85KBy3kq++A8LKQwX6lsu+aWad+2khvuXrqw==", + "dev": true, + "license": "ISC", + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=23" + } + }, + "node_modules/zod": { + "version": "3.25.76", + "resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz", + "integrity": "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/colinhacks" + } + } + } +} diff --git a/labs/converged-modeling-patterns/app/package.json b/labs/converged-modeling-patterns/app/package.json new file mode 100644 index 0000000..d915adc --- /dev/null +++ b/labs/converged-modeling-patterns/app/package.json @@ -0,0 +1,41 @@ +{ + "name": "cmp-lab-ui", + "private": true, + "type": "module", + "engines": { + "node": ">=22" + }, + "scripts": { + "start": "node src/server.js", + "build:vendor": "node build/vendor.mjs", + "test": "vitest run", + "test:unit": "vitest run test/unit", + "test:integration": "vitest run test/integration", + "test:smoke": "node test/smoke/smoke.mjs", + "test:load": "node test/load/load.mjs" + }, + "dependencies": { + "acorn": "^8.18.0", + "express": "^5.2.1", + "mongodb": "^7.7.0", + "oracledb": "^7.0.1", + "yaml": "^2.9.1" + }, + "devDependencies": { + "@codemirror/lang-javascript": "^6.2.5", + "@codemirror/lang-sql": "^6.10.0", + "@codemirror/language": "^6.12.4", + "@codemirror/state": "^6.7.6", + "@codemirror/view": "^6.43.13", + "@fontsource/bricolage-grotesque": "^5.3.0", + "@fontsource/ibm-plex-mono": "^5.3.0", + "@fontsource/public-sans": "^5.3.0", + "@lezer/highlight": "^1.2.5", + "codemirror": "^6.0.2", + "esbuild": "^0.28.2", + "jsdom": "^29.1.1", + "puppeteer-core": "^25.12.0", + "supertest": "^7.3.0", + "vitest": "^5.0.2" + } +} diff --git a/labs/converged-modeling-patterns/app/public/admin.html b/labs/converged-modeling-patterns/app/public/admin.html new file mode 100644 index 0000000..f13a0fc --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/admin.html @@ -0,0 +1,17 @@ + + + + + + +Lab Admin + + + + + +
Lab AdminInstructor
+
+ + + diff --git a/labs/converged-modeling-patterns/app/public/css/lab.css b/labs/converged-modeling-patterns/app/public/css/lab.css new file mode 100644 index 0000000..3bb708c --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/css/lab.css @@ -0,0 +1,110 @@ +/* app/public/css/lab.css */ +:root { + --paper: #F6F8F9; --panel: #FFFFFF; --ink: #1C2733; --muted: #5A6B7C; --faint: #8B99A7; --hairline: #D8DEE4; + --hot: #B3372B; --hot-soft: rgba(179,55,43,.10); --cool: #0E8A63; --cool-soft: rgba(14,138,99,.10); + --flow: #2563EB; --flow-soft: rgba(37,99,235,.10); --chip: #EDF1F4; + --sans: 'Public Sans', 'Segoe UI', system-ui, sans-serif; --head: 'Bricolage Grotesque', var(--sans); --mono: 'IBM Plex Mono', ui-monospace, monospace; +} +@media (prefers-color-scheme: dark) { + :root:not([data-theme="light"]) { + --paper: #131A22; --panel: #1A232E; --ink: #E8EDF2; --muted: #93A3B4; --faint: #6B7B8C; --hairline: #26313D; + --hot: #E06552; --hot-soft: rgba(224,101,82,.14); --cool: #21A57F; --cool-soft: rgba(33,165,127,.14); + --flow: #60A5FA; --flow-soft: rgba(96,165,250,.16); --chip: #222D39; + } +} +:root[data-theme="dark"] { + --paper: #131A22; --panel: #1A232E; --ink: #E8EDF2; --muted: #93A3B4; --faint: #6B7B8C; --hairline: #26313D; + --hot: #E06552; --hot-soft: rgba(224,101,82,.14); --cool: #21A57F; --cool-soft: rgba(33,165,127,.14); + --flow: #60A5FA; --flow-soft: rgba(96,165,250,.16); --chip: #222D39; +} +* { box-sizing: border-box; } +[hidden] { display: none !important; } /* #console sets display: grid, which would otherwise beat the hidden attribute */ +html, body { margin: 0; height: 100%; background: var(--paper); color: var(--ink); font: 15px/1.5 var(--sans); } +header.top { display: flex; align-items: center; gap: 16px; padding: 10px 24px; border-bottom: 1px solid var(--hairline); background: var(--panel); } +header.top .brand { font: 700 15px var(--head); color: var(--ink); text-decoration: none; } +header.top .who { margin-left: auto; font: 12px var(--mono); color: var(--muted); } +main#view { padding: 20px 24px 64px; max-width: 1180px; margin: 0 auto; } +body.dock-pinned main#view { padding-bottom: calc(clamp(280px, 45vh, 60vh) + 24px); } +h1, h2, h3 { font-family: var(--head); margin: 0 0 8px; } +h1 { font-size: 30px; font-weight: 800; } +.kicker { font: 600 12px var(--mono); letter-spacing: .12em; text-transform: uppercase; color: var(--cool); } +.problem { color: var(--muted); max-width: 72ch; } +.knobs { display: flex; flex-wrap: wrap; gap: 8px; margin: 10px 0 14px; } +.knob { font: 12px var(--mono); background: var(--chip); border-radius: 16px; padding: 3px 11px; color: var(--muted); } +.knob b { color: var(--ink); } .knob.hot { background: var(--hot-soft); color: var(--hot); } .knob.hot b { color: var(--hot); } +.patterns { display: grid; grid-template-columns: repeat(auto-fill, minmax(320px, 1fr)); gap: 14px; } +.pcard { display: block; text-decoration: none; color: inherit; background: var(--panel); border: 1px solid var(--hairline); border-left: 4px solid var(--cool); border-radius: 10px; padding: 14px 16px; } +.pcard:hover { border-color: var(--muted); } +.tabs { display: flex; gap: 4px; border-bottom: 1px solid var(--hairline); margin: 14px 0; flex-wrap: wrap; } +.tabs button { font: 600 13px var(--sans); background: none; border: 0; border-bottom: 2px solid transparent; padding: 8px 12px; color: var(--muted); cursor: pointer; } +.tabs button.on { color: var(--ink); border-bottom-color: var(--cool); } +.card { background: var(--panel); border: 1px solid var(--hairline); border-radius: 10px; padding: 12px 14px; margin-bottom: 12px; } +.card h3 { font-size: 15px; font-weight: 700; } +.card .note { color: var(--muted); font-size: 13.5px; margin: 0 0 8px; } +.card.measure { border-left: 3px solid var(--flow); } +pre.code { font: 12.5px/1.5 var(--mono); background: var(--chip); border-radius: 6px; padding: 10px 12px; overflow-x: auto; margin: 0 0 8px; white-space: pre; } +.actions { display: flex; gap: 8px; align-items: center; } +button.btn { font: 600 13px var(--sans); color: var(--ink); background: var(--panel); border: 1px solid var(--hairline); border-radius: 7px; padding: 5px 12px; cursor: pointer; } +button.btn:hover { border-color: var(--muted); } button.btn.primary { background: var(--cool); color: #fff; border-color: var(--cool); } +button.btn:disabled { opacity: .5; cursor: default; } +.badge { font: 600 11px var(--mono); border-radius: 12px; padding: 2px 9px; display: inline-block; margin-bottom: 6px; } +.badge.cool { background: var(--cool-soft); color: var(--cool); } .badge.hot { background: var(--hot-soft); color: var(--hot); } +.result { margin: 6px 0; } .result .msg { font: 13px var(--mono); } .result.error .msg { color: var(--hot); } +.rmeta { font: 11.5px var(--mono); color: var(--faint); margin-top: 4px; } +table.grid { border-collapse: collapse; font: 12.5px var(--mono); width: 100%; } +table.grid th, table.grid td { border-bottom: 1px solid var(--hairline); padding: 4px 8px; text-align: left; vertical-align: top; } +table.grid th { color: var(--faint); font-weight: 600; } +details.jt { font: 12.5px var(--mono); margin-left: 12px; } details.jt > summary { cursor: pointer; color: var(--muted); } +.jv { margin-left: 24px; font: 12.5px var(--mono); } .jk { color: var(--flow); } .jval.string { color: var(--cool); } .jnull { color: var(--faint); } +#console { position: fixed; left: 0; right: 0; bottom: 0; height: clamp(280px, 45vh, 60vh); background: var(--panel); border-top: 1px solid var(--hairline); + display: grid; grid-template-rows: auto 1fr; transform: translateY(calc(100% - 42px)); transition: transform 180ms ease; box-shadow: 0 -6px 18px rgba(0,0,0,.08); z-index: 20; } +#console.expanded { transform: translateY(0); } +#console .dock-toggle, #console .dock-pin { font: 600 12px var(--sans); background: none; border: 0; color: var(--muted); cursor: pointer; padding: 4px 6px; } +#console .dock-pin[aria-pressed="true"] { color: var(--cool); } +#console.busy .dock-toggle::after { content: ' · running'; color: var(--flow); } +@media (prefers-reduced-motion: reduce) { #console { transition: none; } } +#console .bar { display: flex; align-items: center; gap: 8px; padding: 6px 16px; border-bottom: 1px solid var(--hairline); } +#console .bar .tabs { border: 0; margin: 0; } +#console .status { margin-left: auto; font: 12px var(--mono); color: var(--muted); } +#console .panes { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1fr); min-height: 0; } +#console .editor { border-right: 1px solid var(--hairline); overflow: auto; min-height: 0; } +#console .out { overflow: auto; padding: 8px 14px; min-height: 0; } +.cm-editor { height: 100%; font-size: 13px; } .cm-editor .cm-content { font-family: var(--mono); } +.measure-grid { display: grid; grid-template-columns: minmax(0, 3fr) minmax(0, 2fr); gap: 18px; margin-top: 10px; } +.measure-grid.no-curve { grid-template-columns: 1fr; } +@media (max-width: 900px) { .measure-grid { grid-template-columns: 1fr; } } +.measure-out svg.curve { width: 100%; height: auto; display: block; } +.deck-thumb { display: flex; gap: 10px; align-items: center; margin-top: 8px; background: var(--chip); border: 1px solid var(--hairline); border-radius: 8px; padding: 6px; cursor: zoom-in; color: var(--muted); font: 12px var(--sans); text-align: left; width: 100%; } +.deck-thumb img { width: 120px; height: auto; border-radius: 4px; background: var(--paper); } +.stat { display: grid; grid-template-columns: 150px minmax(0, 1fr) 56px; gap: 8px; align-items: center; padding: 6px 0; border-bottom: 1px solid var(--hairline); } +.stat-name { font: 12.5px var(--mono); color: var(--ink); } +.stat-line { display: flex; align-items: center; gap: 6px; height: 16px; } +.stat-bar { display: inline-block; height: 10px; border-radius: 3px; } .stat-bar.hot { background: var(--hot); } .stat-bar.cool { background: var(--cool); } +.stat-val { font: 11.5px var(--mono); color: var(--muted); white-space: nowrap; } +.stat-ratio { font: 700 13px var(--mono); text-align: right; } +.stat-legend { display: flex; gap: 14px; } .stat-legend .key::before { content: ''; display: inline-block; width: 10px; height: 10px; border-radius: 2px; margin-right: 5px; vertical-align: -1px; } +.stat-legend .key.hot::before { background: var(--hot); } .stat-legend .key.cool::before { background: var(--cool); } +.what-happened { margin-top: 14px; background: var(--flow-soft); border-radius: 8px; padding: 10px 14px; font-size: 13.5px; } +.what-happened h4 { margin: 0 0 4px; font: 700 13px var(--head); } .what-happened p { margin: 2px 0; } +.signin { max-width: 420px; margin: 60px auto; background: var(--panel); border: 1px solid var(--hairline); border-radius: 12px; padding: 22px; } +.signin label { display: block; font-size: 13px; color: var(--muted); margin-top: 10px; } +.signin input { width: 100%; font: 15px var(--sans); padding: 7px 10px; border: 1px solid var(--hairline); border-radius: 7px; background: var(--paper); color: var(--ink); } +.signin .btn { margin-top: 16px; } +.flash { color: var(--hot); font-size: 13px; margin-top: 8px; } +@media (max-width: 800px) { #console .panes { grid-template-columns: 1fr; } } +.help-i { font: 13px var(--sans); line-height: 1; background: none; border: 0; color: var(--faint); cursor: pointer; padding: 2px 4px; border-radius: 50%; vertical-align: middle; } +.help-i:hover, .help-i[aria-expanded="true"] { color: var(--flow); } +.help-i:focus-visible { outline: 2px solid var(--flow); outline-offset: 1px; } +.help-pop { position: absolute; z-index: 40; max-width: 420px; max-height: min(70vh, 520px); overflow: auto; background: var(--panel); color: var(--ink); + border: 1px solid var(--hairline); border-left: 3px solid var(--flow); border-radius: 10px; padding: 10px 14px; box-shadow: 0 10px 28px rgba(0,0,0,.18); font-size: 13.5px; } +.help-pop.has-fig { width: min(760px, 92vw); max-width: min(760px, 92vw); max-height: min(88vh, 820px); } +.help-pop p { margin: 2px 0 8px; } +.help-pop .help-title { font: 700 14px var(--head); margin-bottom: 4px; } +.help-pop .help-k { font: 600 10.5px var(--mono); letter-spacing: .1em; text-transform: uppercase; color: var(--muted); } +.help-pop .help-fig { margin: 6px 0 0; } +.help-pop .help-fig a { display: block; } +.help-pop .help-fig img { width: 100%; height: auto; display: block; border-radius: 6px; background: var(--paper); } +.help-pop .help-fig figcaption { font-size: 12px; color: var(--muted); margin-top: 4px; } +.help-pop .help-fig-open { display: inline-block; margin: 2px 0 6px; font: 600 12px var(--sans); color: var(--flow); text-decoration: none; } +.help-pop .help-fig-open:hover { text-decoration: underline; } +.card h3 .help-i, .tabs .help-i, .knob .help-i { margin-left: 4px; } diff --git a/labs/converged-modeling-patterns/app/public/favicon.svg b/labs/converged-modeling-patterns/app/public/favicon.svg new file mode 100644 index 0000000..6181dad --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/favicon.svg @@ -0,0 +1 @@ + diff --git a/labs/converged-modeling-patterns/app/public/help/common.json b/labs/converged-modeling-patterns/app/public/help/common.json new file mode 100644 index 0000000..919712d --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/help/common.json @@ -0,0 +1,6 @@ +{ + "copy": { "what": "Copies the statement to your clipboard.", "why": "Paste it into your own tools or the console below and change it. The lab data is yours to modify." }, + "load": { "what": "Opens the statement in the console at the bottom of the page.", "why": "Edit it before running: change a value, add a predicate, compare plans.", "look": "The console slides up; Ctrl/⌘+Enter runs it." }, + "run": { "what": "Runs the statement in your workspace and shows the result under the card.", "why": "Every run goes through a shared queue so one attendee cannot starve the others; a run has a 10 s budget by default.", "look": "Elapsed time and rows under the result; 'served from cache' means this exact card already ran (by you or a classmate)." }, + "reset": { "what": "Rebuilds this pattern's tables in your workspace to the starting data.", "why": "Use it after you've changed data and want the card results (and Measure it) to match the lab again." } +} diff --git a/labs/converged-modeling-patterns/app/public/index.html b/labs/converged-modeling-patterns/app/public/index.html new file mode 100644 index 0000000..e547bed --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/index.html @@ -0,0 +1,39 @@ + + + + + + +Converged Modeling Lab + + + + + +
+ Converged Modeling Lab + Oracle AI Database 26ai + +
+
+
+
+ +
+ + +
+ + + ready + +
+
+
+ +
+
+
+ + + diff --git a/labs/converged-modeling-patterns/app/public/js/admin.js b/labs/converged-modeling-patterns/app/public/js/admin.js new file mode 100644 index 0000000..6699dde --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/js/admin.js @@ -0,0 +1,105 @@ +// app/public/js/admin.js +import { getJSON, postJSON } from './api.js'; + +const view = document.getElementById('view'); +const h = (tag, cls, text) => { const e = document.createElement(tag); if (cls) e.className = cls; if (text !== undefined) e.textContent = String(text); return e; }; +const btn = (label, onClick, cls = 'btn') => { const b = h('button', cls, label); b.type = 'button'; b.addEventListener('click', onClick); return b; }; +let timer; +// Alert the server's error on a non-200 response; returns true on success. +const ok = (r) => { + if (r.status >= 200 && r.status < 300) return true; + alert(r.body?.error ?? `Request failed (${r.status})`); + return false; +}; + +function login(msg = '') { + const f = h('form', 'signin'); + const l = h('label', null, 'Admin password'); const i = h('input'); i.type = 'password'; i.name = 'password'; l.append(i); + const s = h('button', 'btn primary', 'Sign in'); s.type = 'submit'; + f.append(h('h2', null, 'Instructor sign-in'), l, s, h('div', 'flash', msg)); + f.addEventListener('submit', async (e) => { + e.preventDefault(); + const r = await postJSON('/api/admin/login', { password: i.value }); + if (r.status === 200) render(); else login(r.body?.error ?? 'sign-in failed'); + }); + view.replaceChildren(f); +} + +async function render() { + clearTimeout(timer); + let r; + try { + r = await getJSON('/api/admin/status'); + } catch { + r = { status: 0, body: null }; + } + if (r.status === 401) return login(); + if (r.status === 404) { view.replaceChildren(h('p', 'problem', 'The admin page is only available in event mode (LAB_MODE=event).')); return; } + if (r.status !== 200 || !r.body?.gate) { + // One failed poll (a restart, a busy database) must not stop the page: say so, retry. + const note = h('p', 'flash', `Status unavailable (${r.status ? `HTTP ${r.status}` : 'no response'}), retrying…`); + const old = view.querySelector('p.flash.stale'); + if (old) old.replaceWith(note); else view.prepend(note); + note.classList.add('stale'); + timer = setTimeout(render, 2000); + return; + } + const { gate, cache, attendees, resets, pending } = r.body; + const top = h('div', 'card'); + top.append(h('h3', null, `Queue: ${gate.queued.length} waiting · ${gate.running.length} running · ${gate.paused ? 'PAUSED' : 'live'}`)); + gate.running.forEach((x) => { + const row = h('div', 'actions'); + row.append(h('span', 'rmeta', `${x.userId} · ${x.label} · ${Math.round(x.elapsedMs / 100) / 10}s`), btn('Cancel', async () => { await postJSON('/api/admin/cancel', { id: x.id }); render(); })); + top.append(row); + }); + const controls = h('div', 'actions'); + const n = h('input'); n.type = 'number'; n.min = 1; n.max = 200; n.value = 10; n.style.width = '70px'; + controls.append( + btn(gate.paused ? 'Resume execution' : 'Pause execution', async () => { await postJSON('/api/admin/pause', { on: !gate.paused }); render(); }, 'btn primary'), + btn(`Cache: ${cache.enabled ? 'on' : 'off'} (${cache.size})`, async () => { ok(await postJSON('/api/admin/cache', { enabled: !cache.enabled })); render(); }), + n, btn('Pre-warm workspaces', async () => { ok(await postJSON('/api/admin/prewarm', { count: Number(n.value) })); render(); }), + btn(`Timeouts: ${r.body.timeouts.sqlMs / 1000}s`, async () => { + const raw = prompt('Statement timeout in seconds (1–60)', String(r.body.timeouts.sqlMs / 1000)); + if (raw === null || !/^\s*\d+\s*$/.test(raw)) return; + const v = Number(raw); + if (v < 1 || v > 60) return; + ok(await postJSON('/api/admin/timeouts', { sqlMs: v * 1000, mongoMs: v * 1000 })); render(); + }), + btn('Reset every attendee', async () => { + if (!confirm('Rebuild every pattern in every workspace? This queues behind attendee work.')) return; + ok(await postJSON('/api/admin/reset-attendee', { schema: '*' })); render(); + }), + btn('End event (drop all workspaces)', async () => { + if (prompt('Type END to drop every attendee workspace') !== 'END') return; + const x = await postJSON('/api/admin/end-event', {}); + if (ok(x)) { + const d = x.body?.dropped ?? 0, p = x.body?.pending ?? 0; + alert(p ? `Event ended: ${p} workspaces locked and queued for removal (pending falls to 0 within a few minutes)` : `Event ended: no workspaces to remove`); + } + render(); + }), + ); + top.append(controls); + if (resets && (resets.running || resets.total)) { + const line = resets.running + ? `Resetting every attendee: ${resets.done} of ${resets.total} done` + : `Last reset of every attendee: ${resets.done} of ${resets.total} done`; + top.append(h('p', 'rmeta', `${line} · ${resets.failed.length} failed`)); + resets.failed.forEach((f) => top.append(h('p', 'problem', `${f.schema}: ${f.error}`))); + } + if (pending) top.append(h('p', 'rmeta', `${pending} ended workspace(s) still being dropped (locked; removed automatically)`)); + const t = h('table', 'grid'); + const head = h('tr'); ['Schema', 'Name', 'Email', 'Built patterns', ''].forEach((c) => head.append(h('th', null, c))); + t.append(head); + attendees.forEach((a) => { + const tr = h('tr'); + const td = h('td'); td.append(btn('Reset all patterns', async () => { ok(await postJSON('/api/admin/reset-attendee', { schema: a.schema })); render(); })); + tr.append(h('td', null, a.schema), h('td', null, a.name ?? '(prewarmed)'), h('td', null, a.email ?? ''), h('td', null, a.built.length), td); + t.append(tr); + }); + const list = h('div', 'card'); list.append(h('h3', null, `Attendees (${attendees.length})`), t); + view.replaceChildren(top, list); + timer = setTimeout(render, 2000); +} + +render(); diff --git a/labs/converged-modeling-patterns/app/public/js/api.js b/labs/converged-modeling-patterns/app/public/js/api.js new file mode 100644 index 0000000..fd9bd3a --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/js/api.js @@ -0,0 +1,9 @@ +// app/public/js/api.js +async function call(url, init) { + const res = await fetch(url, { credentials: 'same-origin', ...init }); + let body = null; + try { body = await res.json(); } catch { body = null; } + return { status: res.status, body }; +} +export const getJSON = (url) => call(url); +export const postJSON = (url, body) => call(url, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) }); diff --git a/labs/converged-modeling-patterns/app/public/js/app.js b/labs/converged-modeling-patterns/app/public/js/app.js new file mode 100644 index 0000000..0f961b9 --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/js/app.js @@ -0,0 +1,173 @@ +// app/public/js/app.js +import { getJSON, postJSON } from './api.js'; +import { Console, exec } from './console.js'; +import { Dock } from './dock.js'; +import { renderRun } from './results.js'; +import { renderMeasure } from './measure.js'; +import { helpTrigger } from './help.js'; + +const view = document.getElementById('view'); +const who = document.getElementById('who'); +let config; let patterns = []; let cons; let common = {}; + +const h = (tag, cls, text) => { const e = document.createElement(tag); if (cls) e.className = cls; if (text !== undefined) e.textContent = String(text); return e; }; +const btn = (label, onClick, cls = 'btn') => { const b = h('button', cls, label); b.type = 'button'; b.addEventListener('click', onClick); return b; }; +const MSG = { 409: 'you already have a request queued', 423: 'paused by instructor', 429: 'busy, try again' }; + +async function copy(text, b) { + try { await navigator.clipboard.writeText(text); b.textContent = 'Copied'; } catch { b.textContent = 'Select & copy'; } + setTimeout(() => { b.textContent = 'Copy'; }, 1200); +} + +// One combined ⓘ for the Copy/Load/Run row: each entry is what+why on one line, Run also keeps +// its "look" line (what to check after running). +function cardButtonsHelp() { + const line = (key) => { const c = common[key]; return c ? [c.what, c.why].filter(Boolean).join(' ') : ''; }; + const items = [ + { label: 'Copy', text: line('copy') }, + { label: 'Load into console', text: line('load') }, + { label: 'Run', text: [line('run'), common.run?.look].filter(Boolean).join(' ') }, + ].filter((it) => it.text); + return items.length ? { title: 'The card buttons', items } : null; +} + +function card(title, notes, code, lane, patternId, measureTag, help) { + const c = h('div', 'card'); + if (measureTag) c.append(h('span', 'badge hot', `measured: ${measureTag}`)); + const t = h('h3', null, title); + const hi = helpTrigger(help, { label: title, patternId }); + if (hi) t.append(hi); + c.append(t); + notes.forEach((n) => c.append(h('p', 'note', n))); + c.append(h('pre', 'code', code)); + const out = h('div', 'card-out'); + const copyBtn = btn('Copy', () => copy(code, copyBtn)); + const runBtn = btn('Run', async () => { + runBtn.disabled = true; + const r = await exec({ lane, text: code, patternId }, (s) => { out.textContent = s; }); + runBtn.disabled = false; + out.replaceChildren(r.ok ? renderRun(r.run) : h('div', 'result error', r.message)); + }); + const actions = h('div', 'actions'); + actions.append(copyBtn, btn('Load into console', () => cons.load(lane, code)), runBtn); + const actionsHelp = helpTrigger(cardButtonsHelp(), { label: 'card buttons' }); + if (actionsHelp) actions.append(actionsHelp); + c.append(actions, out); + return c; +} + +function measureCard(p, m) { + const c = h('div', 'card measure'); + const title = h('h3', null, `Measure it · ${m.tag}`); + const th = helpTrigger(p.meta.help.tabs.measure, { label: 'Measure it', patternId: p.id }); if (th) title.append(th); + const kd = h('div', 'kicker', 'Document model'); const kdh = helpTrigger(m.help?.document, { label: 'document write', patternId: p.id }); if (kdh) kd.append(kdh); + const kc = h('div', 'kicker', 'Converged'); const kch = helpTrigger(m.help?.converged, { label: 'converged write', patternId: p.id }); if (kch) kc.append(kch); + c.append(title, kd, h('pre', 'code', m.documentSql), kc, h('pre', 'code', m.convergedSql)); + const out = h('div', 'card-out'); + const b = btn('Measure it', async () => { + b.disabled = true; out.textContent = 'measuring… (waits for an exclusive slot)'; + const r = await postJSON('/api/measure', { patternId: p.id, tag: m.tag }); + b.disabled = false; + out.replaceChildren(r.status === 200 ? renderMeasure(r.body, { pattern: p }) : h('div', 'result error', MSG[r.status] ?? r.body?.error ?? `error ${r.status}`)); + }, 'btn primary'); + c.append(b, out); + return c; +} + +function home() { + cons.setPattern(null); + const grid = h('div', 'patterns'); + patterns.forEach((p, i) => { + const a = h('a', 'pcard'); + a.href = `#/p/${p.id}`; + a.append(h('div', 'kicker', `Pattern ${i + 1} · ${p.meta.industry}`), h('h3', null, p.meta.title), h('p', 'problem', p.meta.problem), h('div', 'rmeta', `Deck slides ${p.meta.deck}`)); + grid.append(a); + }); + view.replaceChildren(h('h1', null, 'Model the domain, not the engine'), + h('p', 'problem', 'Six document-modeling patterns, each with the document-model starting point and the converged alternative. Copy a query, change it, run it in the console below: SQL or the MongoDB API, same data.'), grid); +} + +function patternPage(id) { + const p = patterns.find((x) => x.id === id); + if (!p) return home(); + cons.setPattern(id); + const n = patterns.indexOf(p) + 1; + const head = h('div'); + const knobs = h('div', 'knobs'); + p.meta.knobs.forEach((k) => { const s = h('span', `knob${k.hot ? ' hot' : ''}`); s.append(h('b', null, `${k.name}: `), document.createTextNode(k.setting)); const hi = helpTrigger(k.help ? { why: k.help, figure: { file: 'model-curve.svg', caption: 'The three knobs and where they flip this pattern (deck)' } } : null, { label: k.name, patternId: id }); if (hi) s.append(hi); knobs.append(s); }); + const status = h('span', 'rmeta'); + const reset = btn('Reset this pattern', async () => { + reset.disabled = true; status.textContent = 'resetting…'; + const r = await postJSON('/api/reset', { patternId: id }); + reset.disabled = false; + status.textContent = r.status === 200 ? (r.body.ok ? 'reset done' : `reset failed: ${r.body.errors[0]?.error}`) : (MSG[r.status] ?? `error ${r.status}`); + }); + const resetHelp = helpTrigger(common.reset, { label: 'Reset' }); + head.append(h('div', 'kicker', `Pattern ${n} · ${p.meta.industry} · deck slides ${p.meta.deck}`), h('h1', null, p.meta.title), h('p', 'problem', p.meta.problem), knobs, reset); + if (resetHelp) head.append(resetHelp); + head.append(status); + + const tabs = [ + ['Document model', () => p.cards.document.map((c) => card(c.title, c.notes, c.sql, 'sql', id, c.measure, c.help))], + ['Converged', () => p.cards.converged.map((c) => card(c.title, c.notes, c.sql, 'sql', id, c.measure, c.help))], + ...(p.cards.mongo.length ? [['MongoDB API', () => p.cards.mongo.map((c) => card(c.title, c.notes, c.command, 'mongo', id, null, c.help))]] : []), + ['Measure it', () => p.measures.map((m) => measureCard(p, m))], + ]; + const TABKEY = { 'Document model': 'document', 'Converged': 'converged', 'MongoDB API': 'mongo', 'Measure it': 'measure' }; + const bar = h('div', 'tabs'); + const body = h('div'); + const select = (i) => { + [...bar.querySelectorAll('.tab')].forEach((b, j) => b.classList.toggle('on', i === j)); + body.replaceChildren(...tabs[i][1]()); + }; + tabs.forEach(([label], i) => { + bar.append(btn(label, () => select(i), 'tab')); + const hi = helpTrigger(p.meta.help.tabs[TABKEY[label]], { label, patternId: id }); + if (hi) bar.append(hi); + }); + view.replaceChildren(head, bar, body); + select(0); +} + +function route() { + const m = location.hash.match(/^#\/p\/([\w-]+)/); + if (m) patternPage(m[1]); else home(); + window.scrollTo(0, 0); +} + +function signin() { + const f = h('form', 'signin'); + f.append(h('h2', null, 'Join the lab'), h('p', 'problem', 'Your own workspace is created on first sign-in. Use the same email to come back to it.')); + const field = (label, name, type = 'text') => { const l = h('label', null, label); const i = h('input'); i.name = name; i.type = type; i.required = name !== 'code'; l.append(i); f.append(l); }; + field('Name', 'name'); field('Email', 'email', 'email'); + if (config.eventCodeRequired) field('Event code', 'code'); + const flash = h('div', 'flash'); + const submit = h('button', 'btn primary', 'Sign in'); submit.type = 'submit'; + f.append(submit, flash); + f.addEventListener('submit', async (e) => { + e.preventDefault(); + submit.disabled = true; flash.textContent = 'setting up your workspace…'; + const d = Object.fromEntries(new FormData(f)); + const r = await postJSON('/api/signin', d); + submit.disabled = false; + if (r.status === 200) { flash.textContent = ''; boot(); } else flash.textContent = r.body?.error ?? MSG[r.status] ?? `error ${r.status}`; + }); + document.getElementById('console').hidden = true; + view.replaceChildren(f); +} + +async function boot() { + config = (await getJSON('/api/config')).body; + const me = await getJSON('/api/me'); + if (me.status === 401) return signin(); + document.getElementById('console').hidden = false; + who.textContent = config.mode === 'solo' ? 'solo · CMP_USER' : `${me.body.user.name} · ${me.body.user.schema}`; + patterns = (await getJSON('/api/patterns')).body; + common = (await getJSON('/help/common.json')).body ?? {}; + const consoleEl = document.getElementById('console'); + cons ??= new Console(consoleEl, new Dock(consoleEl)); + window.onhashchange = route; + route(); +} + +boot(); diff --git a/labs/converged-modeling-patterns/app/public/js/chart.js b/labs/converged-modeling-patterns/app/public/js/chart.js new file mode 100644 index 0000000..f403853 --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/js/chart.js @@ -0,0 +1,155 @@ +// Log-log "write amplification vs size" chart: the measured reference calibration as a line with +// points, y = 1 (break-even) dashed, and this attendee's live measurement as a marker at lab_x. +// Label placement is collision-aware (estimated text boxes, not exact metrics) because the same +// chart has to work across every pattern's calibration data, not just the ones checked by hand. +const NS = 'http://www.w3.org/2000/svg'; +const s = (tag, attrs = {}, text) => { const e = document.createElementNS(NS, tag); for (const [k, v] of Object.entries(attrs)) e.setAttribute(k, String(v)); if (text != null) e.textContent = String(text); return e; }; +const finite = (n) => typeof n === 'number' && Number.isFinite(n) && n > 0; +const fmtN = (v) => (v >= 1000 ? `${v / 1000}k` : String(v)); + +export function logScale([d0, d1], [r0, r1]) { + const l0 = Math.log10(d0); const l1 = Math.log10(d1); + const f = (v) => r0 + ((Math.log10(v) - l0) / (l1 - l0)) * (r1 - r0); + f.ticks = () => { const out = []; for (let p = Math.ceil(l0); p <= Math.floor(l1); p++) out.push(10 ** p); return out; }; + return f; +} + +// Rough text-box estimate (0.6 * font-size per character — no real metrics available at this +// layer) used only to decide "does this collide", not to render. +const charW = (fs, str) => 0.6 * fs * str.length; +const boxFor = (cx, baseline, anchor, str, fs) => { + const w = charW(fs, str); + const x0 = anchor === 'end' ? cx - w : anchor === 'middle' ? cx - w / 2 : cx; + return { x0, x1: x0 + w, y0: baseline - fs, y1: baseline + fs * 0.3 }; +}; +const overlaps = (a, b) => a.x0 < b.x1 && a.x1 > b.x0 && a.y0 < b.y1 && a.y1 > b.y0; + +export function curveChart({ calibration, labX, xLabel, live }) { + const W = 560; const H = 300; const L = 64; const R = 20; const T = 22; const B = 50; + const R_REDO = 7; const R_BLOCKS = 6; + const xs = calibration.map((c) => c.x); const ys = calibration.map((c) => c.ratio); + const yl = [live?.redo, live?.blocks].filter(finite); + const x = logScale([Math.min(...xs) / 1.5, Math.max(...xs) * 1.5], [L, W - R]); + const yMin = Math.min(0.5, ...ys, ...yl) / 1.3; const yMax = Math.max(...ys, ...yl) * 1.5; + const y = logScale([yMin, yMax], [H - B, T]); + const svg = s('svg', { viewBox: `0 0 ${W} ${H}`, class: 'curve', role: 'img', 'aria-label': `Document model divided by converged, versus ${xLabel}` }); + + // Every dynamic label in the plot gets a halo so it stays legible crossing gridlines or the + // calibration curve (static ticks/titles get it too — harmless over the plain paper background). + const t = (attrs, text) => s('text', { 'paint-order': 'stroke', stroke: 'var(--panel)', 'stroke-width': 3, ...attrs }, text); + + svg.append(s('line', { x1: L, y1: H - B, x2: W - R, y2: H - B, stroke: 'var(--hairline)' })); + svg.append(s('line', { x1: L, y1: T, x2: L, y2: H - B, stroke: 'var(--hairline)' })); + for (const tk of x.ticks()) { + svg.append(s('line', { x1: x(tk), y1: T, x2: x(tk), y2: H - B, stroke: 'var(--hairline)', 'stroke-dasharray': '2 4' })); + svg.append(t({ x: x(tk), y: H - B + 16, 'text-anchor': 'middle', 'font-size': 11, fill: 'var(--muted)' }, fmtN(tk))); + } + for (const tk of y.ticks()) { + svg.append(t({ x: L - 8, y: y(tk) + 4, 'text-anchor': 'end', 'font-size': 11, fill: 'var(--muted)' }, `${tk}×`)); + } + svg.append(t({ x: (L + W - R) / 2, y: H - 10, 'text-anchor': 'middle', 'font-size': 12, fill: 'var(--ink)' }, `${xLabel} (log)`)); + svg.append(t({ x: 14, y: (T + H - B) / 2, 'text-anchor': 'middle', 'font-size': 12, fill: 'var(--ink)', transform: `rotate(-90 14 ${(T + H - B) / 2})` }, 'document ÷ converged (log)')); + + // Calibration points (r 3.5) padded by 4px: every label placed below treats them as obstacles. + const CAL_R = 3.5; + const calObstacles = calibration.map((c) => ({ x0: x(c.x) - CAL_R - 4, x1: x(c.x) + CAL_R + 4, y0: y(c.ratio) - CAL_R - 4, y1: y(c.ratio) + CAL_R + 4 })); + + // Boxes already committed to the plot — break-even, lab-data, then the live labels as each is + // placed — so later placements know what to avoid. + const placed = []; + + if (yMin < 1 && yMax > 1) { + svg.append(s('line', { class: 'breakeven', x1: L, y1: y(1), x2: W - R, y2: y(1), stroke: 'var(--muted)', 'stroke-dasharray': '6 4' })); + const beText = 'break-even · above = document writes more'; const beFs = 11; + // Try left-above, right-above, left-below, right-below the line; take the first that stays + // 10px clear of the lab-data guide and clear of every calibration point. A point at small x + // with a ratio just above 1 (02-computed's x=10) sits exactly where left-above would go. + const labStrip = { x0: x(labX) - 10, x1: x(labX) + 10, y0: T, y1: H - B }; + const cands = []; + for (const ty of [y(1) - 6, y(1) + beFs + 4]) { + for (const [tx, anchor] of [[L + 4, 'start'], [W - R - 4, 'end']]) cands.push({ tx, ty, anchor, box: boxFor(tx, ty, anchor, beText, beFs) }); + } + const be = cands.find((c) => ![labStrip, ...calObstacles].some((o) => overlaps(c.box, o))) ?? cands[0]; + svg.append(t({ x: be.tx, y: be.ty, 'text-anchor': be.anchor, 'font-size': beFs, fill: 'var(--muted)' }, beText)); + placed.push(be.box); + } + + const pts = calibration.map((c) => `${x(c.x).toFixed(1)},${y(c.ratio).toFixed(1)}`); + svg.append(s('path', { class: 'cal', d: `M${pts.join(' L')}`, fill: 'none', stroke: 'var(--hot)', 'stroke-width': 2.2 })); + for (const c of calibration) svg.append(s('circle', { class: 'calpt', cx: x(c.x), cy: y(c.ratio), r: CAL_R, fill: 'var(--panel)', stroke: 'var(--hot)', 'stroke-width': 1.6 })); + // The "measured on 26ai Free (reference run)" annotation used to live here, pinned beside the + // highest calibration point — which is also always near the top of the chart, since every + // pattern's ratio increases with size. That collided with the lab-data label whenever the lab's + // own x was log-close to the domain's high end. It now lives in the caption below the chart + // (measure.js), which needs no collision handling at all. + + // ---- live markers ---- + const redoY = finite(live?.redo) ? y(live.redo) : null; + const blocksY = finite(live?.blocks) ? y(live.blocks) : null; + const dist = (ax, ay, bx, by) => Math.hypot(ax - bx, ay - by); + const nearCalibration = (cy, r) => calibration.some((c) => dist(x(labX), cy, x(c.x), y(c.ratio)) < r); + // The lab's x is often exactly one of the calibration x's by construction, so a live ratio close + // to that point's reference ratio lands the marker almost on top of the calibration circle — and + // the two live markers can just as easily land on top of each other. Either case: split them + // apart horizontally rather than let them merge into one blob. + let needOffset = false; + if (redoY != null && blocksY != null && Math.abs(redoY - blocksY) < R_REDO + R_BLOCKS + 2) needOffset = true; + if (redoY != null && nearCalibration(redoY, R_REDO)) needOffset = true; + if (blocksY != null && nearCalibration(blocksY, R_BLOCKS)) needOffset = true; + const redoCx = x(labX) + (needOffset ? 8 : 0); + const blocksCx = x(labX) - (needOffset ? 8 : 0); + + // The guide line used to run the full plot height; now it stops short of the higher marker (or + // at the top when there is none) so it never runs through a label sitting above it. + const markerYs = [redoY, blocksY].filter((v) => v != null); + const guideTop = markerYs.length ? Math.max(T, Math.min(...markerYs) - 12) : T; + svg.append(s('line', { x1: x(labX), y1: guideTop, x2: x(labX), y2: H - B, stroke: 'var(--flow)', 'stroke-dasharray': '3 3' })); + const labFs = 11; const labText = `lab data · ${fmtN(labX)}`; + svg.append(t({ x: x(labX) + 4, y: H - B - 6, 'font-size': labFs, fill: 'var(--flow)' }, labText)); + placed.push(boxFor(x(labX) + 4, H - B - 6, 'start', labText, labFs)); + + // Greedy label placement: try right of the marker, then left, then above, then below; take the + // first whose estimated box clears every marker, calibration point and label already placed. Clamped inside + // [L, W-R]; a "right" try that would overflow past the edge instead anchors from the marker. + function placeLabel(mx, my, r, str, fs, obstacles) { + const gap = 6; + const clampX = (tx, anchor) => { + let box = boxFor(tx, my, anchor, str, fs); + if (box.x0 < L) tx += L - box.x0; + box = boxFor(tx, my, anchor, str, fs); + if (box.x1 > W - R) tx -= box.x1 - (W - R); + return tx; + }; + let right = { x: mx + r + gap, y: my + fs * 0.32, anchor: 'start' }; + let rightBox = boxFor(right.x, right.y, right.anchor, str, fs); + if (rightBox.x1 > W - R) { right = { x: mx - 12, y: right.y, anchor: 'end' }; rightBox = boxFor(right.x, right.y, right.anchor, str, fs); } + const leftX = clampX(mx - r - gap, 'end'); const left = { x: leftX, y: my + fs * 0.32, anchor: 'end', box: boxFor(leftX, my + fs * 0.32, 'end', str, fs) }; + const aboveX = clampX(mx, 'middle'); const above = { x: aboveX, y: my - r - gap, anchor: 'middle', box: boxFor(aboveX, my - r - gap, 'middle', str, fs) }; + const belowY = my + r + gap + fs; const below = { x: aboveX, y: belowY, anchor: 'middle', box: boxFor(aboveX, belowY, 'middle', str, fs) }; + for (const cand of [{ ...right, box: rightBox }, left, above, below]) { + if (!obstacles.some((o) => overlaps(cand.box, o))) return cand; + } + return { ...right, box: rightBox }; + } + + if (redoY != null) { + const markerObstacles = blocksY != null ? [{ x0: blocksCx - R_BLOCKS, x1: blocksCx + R_BLOCKS, y0: blocksY - R_BLOCKS, y1: blocksY + R_BLOCKS }] : []; + const text = `you · ${live.redo.toFixed(1)}× redo`; + const label = placeLabel(redoCx, redoY, R_REDO, text, 12, [...placed, ...calObstacles, ...markerObstacles]); + svg.append(t({ x: label.x, y: label.y, 'text-anchor': label.anchor, 'font-size': 12, 'font-weight': 700, fill: 'var(--flow)' }, text)); + placed.push(label.box); + } + if (blocksY != null) { + const markerObstacles = redoY != null ? [{ x0: redoCx - R_REDO, x1: redoCx + R_REDO, y0: redoY - R_REDO, y1: redoY + R_REDO }] : []; + const text = `blocks ${live.blocks.toFixed(1)}×`; + const label = placeLabel(blocksCx, blocksY, R_BLOCKS, text, 11, [...placed, ...calObstacles, ...markerObstacles]); + svg.append(t({ x: label.x, y: label.y, 'text-anchor': label.anchor, 'font-size': 11, fill: 'var(--flow)' }, text)); + placed.push(label.box); + } + + // Markers drawn last so they sit on top of the calibration curve, points and gridlines. + if (blocksY != null) svg.append(s('circle', { class: 'live-blocks', cx: blocksCx, cy: blocksY, r: R_BLOCKS, fill: 'none', stroke: 'var(--flow)', 'stroke-width': 2 })); + if (redoY != null) svg.append(s('circle', { class: 'live-redo', cx: redoCx, cy: redoY, r: R_REDO, fill: 'var(--flow)' })); + + return svg; +} diff --git a/labs/converged-modeling-patterns/app/public/js/console.js b/labs/converged-modeling-patterns/app/public/js/console.js new file mode 100644 index 0000000..50705f5 --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/js/console.js @@ -0,0 +1,101 @@ +// app/public/js/console.js +import { createEditor } from '/vendor/cm.js'; +import { getJSON, postJSON } from './api.js'; +import { renderRun } from './results.js'; + +const MESSAGES = { 401: 'please sign in again', 409: 'you already have a request queued', 423: 'paused by instructor', 429: 'busy: the database is serving others, try again' }; +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const store = { + get(k) { try { return JSON.parse(localStorage.getItem(k)); } catch { return null; } }, + set(k, v) { try { localStorage.setItem(k, JSON.stringify(v)); } catch { /* private mode */ } }, +}; + +export async function exec({ lane, text, patternId }, onStatus) { + let done = false; + onStatus('running…'); + const poll = (async () => { + await sleep(400); + while (!done) { + const q = await getJSON('/api/queue').catch(() => null); + if (!done && q?.status === 200 && q.body) onStatus(q.body.paused ? 'paused by instructor' : q.body.position === null ? 'running…' : `queued · ${q.body.position} ahead`); + await sleep(750); + } + })(); + try { + const r = await postJSON('/api/run', { lane, patternId, text }); + if (r.status === 200) return { ok: true, run: r.body }; + return { ok: false, message: MESSAGES[r.status] ?? r.body?.error ?? `error ${r.status}` }; + } catch { + return { ok: false, message: 'network error' }; + } finally { + done = true; + await poll.catch(() => {}); + } +} + +export class Console { + constructor(root, dock = null) { + this.root = root; + this.dock = dock; + this.lane = 'sql'; + this.patternId = null; + this.statusEl = root.querySelector('#status'); + this.out = root.querySelector('#out'); + this.editors = { + sql: createEditor(root.querySelector('#editor-sql'), { lang: 'sql', doc: store.get('lab.last.sql') ?? 'SELECT 1 FROM dual', onRun: () => this.run() }), + mongo: createEditor(root.querySelector('#editor-mongo'), { lang: 'mongo', doc: store.get('lab.last.mongo') ?? 'show collections', onRun: () => this.run() }), + }; + root.querySelectorAll('.bar [data-lane]').forEach((b) => b.addEventListener('click', () => this.show(b.dataset.lane))); + root.querySelector('#run').addEventListener('click', () => this.run()); + root.querySelector('#history').addEventListener('click', () => this.showHistory()); + } + + show(lane) { + this.lane = lane; + this.root.querySelectorAll('.bar [data-lane]').forEach((b) => b.classList.toggle('on', b.dataset.lane === lane)); + this.root.querySelector('#editor-sql').hidden = lane !== 'sql'; + this.root.querySelector('#editor-mongo').hidden = lane !== 'mongo'; + this.editors[lane].focus(); + } + + load(lane, text) { + this.dock?.open('load'); + this.show(lane); + this.editors[lane].set(text); + this.editors[lane].focus(); + } + + setPattern(id) { + this.patternId = id; + } + + async run() { + const lane = this.lane; + const text = this.editors[lane].get(); + if (!text.trim()) return; + store.set(`lab.last.${lane}`, text); + const hist = (store.get(`lab.hist.${lane}`) ?? []).filter((t) => t !== text); + store.set(`lab.hist.${lane}`, [text, ...hist].slice(0, 20)); + const btn = this.root.querySelector('#run'); + btn.disabled = true; + this.dock?.setBusy(true); + const r = await exec({ lane, text, patternId: this.patternId }, (s) => { this.statusEl.textContent = s; }); + this.dock?.setBusy(false); + btn.disabled = false; + this.statusEl.textContent = r.ok ? 'ready' : r.message; + this.out.replaceChildren(r.ok ? renderRun(r.run) : Object.assign(document.createElement('div'), { className: 'result error', textContent: r.message })); + } + + showHistory() { + const list = document.createElement('div'); + const items = store.get(`lab.hist.${this.lane}`) ?? []; + if (!items.length) list.textContent = 'No history yet.'; + for (const t of items) { + const pre = Object.assign(document.createElement('pre'), { className: 'code', textContent: t, title: 'Click to load' }); + pre.style.cursor = 'pointer'; + pre.addEventListener('click', () => this.load(this.lane, t)); + list.append(pre); + } + this.out.replaceChildren(list); + } +} diff --git a/labs/converged-modeling-patterns/app/public/js/dock.js b/labs/converged-modeling-patterns/app/public/js/dock.js new file mode 100644 index 0000000..f966826 --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/js/dock.js @@ -0,0 +1,89 @@ +// Console dock: a 40 px bar that slides the console up on hover, focus, load or run, and back +// down 600 ms after the pointer leaves, or after focus moves outside the dock, or after a run +// finishes — whichever happens last while the pointer isn't on it — unless the dock is pinned +// or another one of those stays true. Touch devices (no hover) toggle on tap only. +const OPEN_DELAY = 150; +const CLOSE_DELAY = 600; +const KEY = 'lab.dock.pinned'; + +export class Dock { + constructor(root, { storage = globalThis.localStorage } = {}) { + this.root = root; + this.storage = storage; + this.toggle = root.querySelector('#dock-toggle'); + this.pinBtn = root.querySelector('#dock-pin'); + this.focused = false; this.busy = false; this.hovered = false; this.timer = null; + this.hoverable = globalThis.matchMedia?.('(hover: hover)').matches ?? true; + this.#setPinned(this.#read() === 'true'); + this.#render(this.pinned); + // hovered is tracked unconditionally — a run or Load can open the dock on a touch + // device too, and closing still needs to know the pointer isn't sitting on it. + root.addEventListener('pointerenter', () => { + this.hovered = true; + if (this.hoverable) this.#schedule(() => this.open('hover'), OPEN_DELAY); + }); + root.addEventListener('pointerleave', () => { + this.hovered = false; + if (this.hoverable) this.#schedule(() => this.close(), CLOSE_DELAY); + }); + // Focus on the toggle or pin must not open the dock: Esc returns focus to the toggle (which + // would reopen what Esc just closed), and a tap focuses then clicks (which would open, then + // immediately close). The toggle's own click is the only thing that opens it from there. + root.addEventListener('focusin', (e) => { + this.focused = true; + if (e.target !== this.toggle && e.target !== this.pinBtn) this.open('focus'); + }); + root.addEventListener('focusout', (e) => { + if (!root.contains(e.relatedTarget)) { + this.focused = false; + // Load/Run open the dock via focus, not hover — nothing else ever closed it again. + if (!this.hovered) this.#schedule(() => this.close(), CLOSE_DELAY); + } + }); + root.addEventListener('keydown', (e) => { if (e.key === 'Escape' && !this.pinned) { this.focused = false; this.close(true); this.toggle?.focus(); } }); + this.toggle?.addEventListener('click', () => (this.expanded ? this.close(true) : this.open('toggle'))); + this.pinBtn?.addEventListener('click', () => this.togglePin()); + globalThis.addEventListener?.('keydown', (e) => { if ((e.ctrlKey || e.metaKey) && e.key === '`') { e.preventDefault(); this.expanded ? this.close(true) : this.open('key'); } }); + } + + get expanded() { return this.root.classList.contains('expanded'); } + + open() { clearTimeout(this.timer); this.#render(true); } + + close(force = false) { + clearTimeout(this.timer); + if (this.pinned || this.busy || (this.focused && !force)) return; + this.#render(false); + } + + setBusy(on) { + this.busy = on; + this.root.classList.toggle('busy', on); + if (on) this.open('run'); + else if (!this.hovered && !this.focused) this.#schedule(() => this.close(), CLOSE_DELAY); + } + + togglePin() { + this.#setPinned(!this.pinned); + try { this.storage?.setItem(KEY, String(this.pinned)); } catch { /* storage unavailable */ } + if (this.pinned) this.open('pin'); + } + + #setPinned(v) { + this.pinned = v; + this.root.classList.toggle('pinned', v); + // A pinned dock covers the bottom of the page; the body class lets CSS pad main#view for it. + this.root.ownerDocument?.body?.classList.toggle('dock-pinned', v); + this.pinBtn?.setAttribute('aria-pressed', String(v)); + } + + #read() { try { return this.storage?.getItem(KEY); } catch { return null; } } + + #schedule(fn, ms) { clearTimeout(this.timer); this.timer = setTimeout(fn, ms); } + + #render(on) { + this.root.classList.toggle('expanded', on); + this.toggle?.setAttribute('aria-expanded', String(on)); + if (this.toggle) this.toggle.textContent = on ? '▼ Console' : '▲ Console'; + } +} diff --git a/labs/converged-modeling-patterns/app/public/js/help.js b/labs/converged-modeling-patterns/app/public/js/help.js new file mode 100644 index 0000000..df4804a --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/js/help.js @@ -0,0 +1,126 @@ +// ⓘ help: a small button that opens one popover at a time with "What it does / Why it matters / +// What to look for" (or a labelled list of items) and an optional deck figure — the popover widens +// (.has-fig) and the figure links to the full-size file when one is shown. Hover opens a tooltip +// (desktop) that describes its trigger (aria-describedby) and closes on Esc from anywhere; click, +// Enter or Space opens a dialog that keeps focus until Esc or an outside click. +// Text via textContent only. +const GUTTER = 16; +const GAP = 8; +// Grace period for a hover tooltip: long enough to cross the gap from the ⓘ into the +// popover (and reach its links), short enough that it still feels like hover. +const LEAVE_GRACE = 350; +let leaveTimer = null; +const cancelLeave = () => { clearTimeout(leaveTimer); leaveTimer = null; }; +const scheduleLeave = () => { cancelLeave(); leaveTimer = setTimeout(() => { if (current && !current.modal) closeCurrent(); }, LEAVE_GRACE); }; +let current = null; +let seq = 0; + +const h = (tag, cls, text) => { const e = document.createElement(tag); if (cls) e.className = cls; if (text != null) e.textContent = String(text); return e; }; + +export function placePopover(rect, size, vp) { + const below = rect.bottom + GAP; + const side = below + size.height <= vp.height - GUTTER || rect.top - GAP - size.height < GUTTER ? 'below' : 'above'; + const top = side === 'below' ? below : rect.top - GAP - size.height; + const left = Math.max(GUTTER, Math.min(rect.left, vp.width - GUTTER - size.width)); + return { top, left, side }; +} + +function closeCurrent(returnFocus = false) { + cancelLeave(); + if (!current) return; + const { pop, trigger } = current; + pop.remove(); + trigger.setAttribute('aria-expanded', 'false'); + trigger.removeAttribute('aria-describedby'); + current = null; + if (returnFocus) trigger.focus(); +} + +function build(content, { label, patternId }, modal) { + const pop = h('div', 'help-pop'); + pop.id = `help-pop-${++seq}`; + pop.setAttribute('role', modal ? 'dialog' : 'tooltip'); + pop.setAttribute('aria-label', `About: ${label}`); + pop.tabIndex = -1; + if (content.title) pop.append(h('div', 'help-title', content.title)); + content.items?.forEach((it) => pop.append(h('div', 'help-k', it.label), h('p', null, it.text))); + if (content.what) pop.append(h('div', 'help-k', 'What it does'), h('p', null, content.what)); + if (content.why) pop.append(h('div', 'help-k', 'Why it matters'), h('p', null, content.why)); + if (content.look) pop.append(h('div', 'help-k', 'What to look for'), h('p', null, content.look)); + if (content.figure && patternId) { + pop.classList.add('has-fig'); + const href = `/figures/${patternId}/${content.figure.file}`; + const fig = h('figure', 'help-fig'); + const link = h('a'); + link.href = href; + link.target = '_blank'; + link.rel = 'noopener'; + const img = h('img'); + img.src = href; + img.alt = content.figure.caption ?? label; + img.loading = 'lazy'; + link.append(img); + // The full-size link goes ABOVE the figure: a tall figure pushes anything below it + // out of the popover's visible area, where a hover tooltip can't be scrolled to. + const open = h('a', 'help-fig-open', 'Open full size ↗'); + open.href = href; + open.target = '_blank'; + open.rel = 'noopener'; + fig.append(open, link); + if (content.figure.caption) fig.append(h('figcaption', null, content.figure.caption)); + pop.append(fig); + } + return pop; +} + +function open(trigger, content, opts, modal) { + if (current?.trigger === trigger && current.modal === modal) return; + closeCurrent(); + const pop = build(content, opts, modal); + document.body.append(pop); + const r = trigger.getBoundingClientRect(); + const size = { width: pop.offsetWidth || 420, height: pop.offsetHeight || 200 }; + const p = placePopover(r, size, { width: window.innerWidth, height: window.innerHeight }); + pop.style.top = `${p.top + window.scrollY}px`; + pop.style.left = `${p.left + window.scrollX}px`; + pop.dataset.side = p.side; + pop.addEventListener('keydown', (e) => { if (e.key === 'Escape') { e.stopPropagation(); closeCurrent(true); } }); + pop.addEventListener('pointerenter', () => { if (!modal) cancelLeave(); }); + pop.addEventListener('pointerleave', () => { if (!modal) scheduleLeave(); }); + trigger.setAttribute('aria-expanded', 'true'); + if (!modal) trigger.setAttribute('aria-describedby', pop.id); + current = { pop, trigger, modal }; + if (modal) pop.focus(); +} + +document.addEventListener?.('pointerdown', (e) => { + if (current && !current.pop.contains(e.target) && e.target !== current.trigger) closeCurrent(); +}); + +// A hover tooltip never takes focus, so its own keydown never fires — Esc is caught here instead. +document.addEventListener?.('keydown', (e) => { + if (e.key === 'Escape' && current && !current.modal) closeCurrent(); +}); + +// Open a dialog popover anchored to any element (e.g. the Measure-it deck thumbnail). +export function openHelp(anchor, content, opts) { open(anchor, content, opts, true); } + +export function helpTrigger(content, opts) { + if (!content || !(content.why || content.look || content.what || content.figure || content.items?.length)) return null; + const b = h('button', 'help-i', 'ⓘ'); + b.type = 'button'; + b.setAttribute('aria-label', `About: ${opts.label}`); + b.setAttribute('aria-haspopup', 'dialog'); + b.setAttribute('aria-expanded', 'false'); + let hoverTimer = null; + b.addEventListener('click', (e) => { e.stopPropagation(); open(b, content, opts, true); }); + b.addEventListener('pointerenter', () => { + if (current?.trigger === b && !current.modal) { cancelLeave(); return; } + hoverTimer = setTimeout(() => open(b, content, opts, false), 200); + }); + b.addEventListener('pointerleave', (e) => { + clearTimeout(hoverTimer); + if (current?.trigger === b && !current.modal && !current.pop.contains(e.relatedTarget)) scheduleLeave(); + }); + return b; +} diff --git a/labs/converged-modeling-patterns/app/public/js/measure.js b/labs/converged-modeling-patterns/app/public/js/measure.js new file mode 100644 index 0000000..f6c7b84 --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/js/measure.js @@ -0,0 +1,89 @@ +// Measure-it result: the calibrated curve with this run's live point, paired bars per statistic +// (each row scaled to its own max) with ratios and ⓘ explainers, and a "what just happened" panel. +// If either side errored there is no live point and every ratio shows — (raw values still shown). +import { curveChart } from './chart.js'; +import { helpTrigger, openHelp } from './help.js'; + +const STATS = [ + ['redo size', 'Bytes the database must write to its log to make the change durable: the most direct measure of how much the write physically changed.'], + ['db block changes', 'How many data blocks the write touched.'], + ['session logical reads', 'Blocks read (from cache) to find and change the data.'], + ['CPU used by this session', 'CPU time in centiseconds. At this data size it is usually 0 on both sides. Shown, not hidden.'], +]; +const num = (n) => (typeof n === 'number' && Number.isFinite(n) ? n : null); +const fmt = (n) => (num(n) === null ? '—' : n.toLocaleString('en-US')); +const ratioOf = (d, c) => (num(d) === null || !num(c) ? null : d / c); +const fmtRatio = (r) => (r === null ? '—' : `${r.toFixed(1)}×`); +const h = (tag, cls, text) => { const e = document.createElement(tag); if (cls) e.className = cls; if (text != null) e.textContent = String(text); return e; }; + +function statBars(m, failed) { + const wrap = h('div', 'stat-bars'); + for (const [name, explain] of STATS) { + const d = m.document.stats[name]; const c = m.converged.stats[name]; + // Each row scales to its own max, not a scale shared across all four stats — a shared scale + // (even a log one) draws a 33x gap as if it were roughly 1.5x once other rows have much bigger + // numbers. Zero gets no bar at all, not a sliver. + const max = Math.max(num(d) ?? 0, num(c) ?? 0); + const len = (v) => (v > 0 && max > 0 ? Math.max(1.5, (v / max) * 100) : 0); + const row = h('div', 'stat'); + const label = h('div', 'stat-name', name); + const hi = helpTrigger({ what: explain }, { label: name }); if (hi) label.append(hi); + const bars = h('div', 'stat-pair'); + for (const [v, cls] of [[d, 'hot'], [c, 'cool']]) { + const line = h('div', 'stat-line'); + const bar = h('span', `stat-bar ${cls}`); bar.style.width = `${len(num(v) ?? 0)}%`; + line.append(bar, h('span', 'stat-val', fmt(v))); + bars.append(line); + } + row.append(label, bars, h('div', 'stat-ratio', fmtRatio(failed ? null : ratioOf(d, c)))); + wrap.append(row); + } + const rows = (r) => (r.kind === 'dml' ? r.rowsAffected : null); + wrap.append(h('div', 'rows-affected rmeta', + `rows affected ${fmt(rows(m.document.result))} vs ${fmt(rows(m.converged.result))} · elapsed ${fmt(m.document.result.elapsedMs)} ms vs ${fmt(m.converged.result.elapsedMs)} ms · bars scaled per row`)); + const legend = h('div', 'stat-legend rmeta'); + legend.append(h('span', 'key hot', 'document model'), h('span', 'key cool', 'converged')); + wrap.prepend(legend); + return wrap; +} + +function whatHappened() { + const box = h('div', 'what-happened'); + box.append(h('h4', null, 'What just happened')); + for (const t of [ + 'Both writes ran back to back in one exclusive slot, so nobody else\'s work is in these numbers.', + 'Each side ran once unmeasured as a warm-up, then once measured, then rolled back. Your lab data is unchanged.', + 'Read the ratio, not the bytes: the bytes depend on this small lab dataset; the curve shows how the gap scales with the size on the x-axis; your point shows where this lab sits on it.', + 'One session on Oracle AI Database 26ai Free: expect small run-to-run variance (a block, or a LOB chunk). Say "same order of magnitude", not exact bytes.', + ]) box.append(h('p', null, t)); + return box; +} + +export function renderMeasure(m, { pattern } = {}) { + const box = h('div', 'measure-out'); + for (const [name, sd] of [['document model', m.document], ['converged', m.converged]]) { + if (sd.result.kind === 'error') box.append(h('div', 'result error', `${name}: ${sd.result.code ?? ''} ${sd.result.error}`)); + } + // A failed side's stats describe a partial write — no ratio of it means anything. + const failed = m.document.result.kind === 'error' || m.converged.result.kind === 'error'; + const meas = pattern?.meta?.measure; + const grid = h('div', meas ? 'measure-grid' : 'measure-grid no-curve'); + if (meas) { + const left = h('div', 'measure-curve'); + left.append(h('div', 'kicker', 'Write amplification vs size')); + left.append(curveChart({ calibration: meas.calibration, labX: meas.labX, xLabel: meas.xLabel, + live: failed ? null : { redo: ratioOf(m.document.stats['redo size'], m.converged.stats['redo size']), blocks: ratioOf(m.document.stats['db block changes'], m.converged.stats['db block changes']) } })); + left.append(h('p', 'rmeta', 'Solid dot: your redo ratio · ring: your block-change ratio · line: reference measurements on 26ai Free at other sizes, same protocol.')); + const thumb = h('button', 'deck-thumb'); thumb.type = 'button'; + const img = h('img'); img.src = `/figures/${pattern.id}/model-curve.svg`; img.alt = 'The deck\'s workload model for this pattern'; + thumb.append(img, h('span', null, `The full workload picture: illustrative model from the deck (slides ${meas.deckSlides})`)); + thumb.addEventListener('click', () => openHelp(thumb, { title: 'Workload model (deck)', what: 'The deck\'s daily cost model: reads plus writes, in illustrative units, across the pattern\'s knob. The live chart is the part Measure it can prove: one write, measured as the ratio of redo between the two models.', figure: { file: 'model-curve.svg', caption: `Deck slides ${meas.deckSlides}` } }, { label: 'workload model', patternId: pattern.id })); + left.append(thumb); + grid.append(left); + } + const right = h('div', 'measure-stats'); + right.append(h('div', 'kicker', 'This run'), statBars(m, failed)); + grid.append(right); + box.append(grid, whatHappened()); + return box; +} diff --git a/labs/converged-modeling-patterns/app/public/js/results.js b/labs/converged-modeling-patterns/app/public/js/results.js new file mode 100644 index 0000000..ba76899 --- /dev/null +++ b/labs/converged-modeling-patterns/app/public/js/results.js @@ -0,0 +1,87 @@ +// app/public/js/results.js +const h = (tag, cls, text) => { + const el = document.createElement(tag); + if (cls) el.className = cls; + if (text !== undefined) el.textContent = String(text); + return el; +}; + +// node-oracledb DATE/TIMESTAMP values come back through res.json() as ISO-8601 +// strings (e.g. "2026-01-10T09:00:00.000Z"), not Date objects. Reformat those +// for readability; anything that isn't a full ISO timestamp is left untouched. +const ISO_TS = /^(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2})(\.(\d+))?Z$/; +function formatTimestamp(s) { + const m = ISO_TS.exec(s); + if (!m) return null; + const [, date, time, , frac] = m; + const keepFrac = frac && /[1-9]/.test(frac) ? `.${frac}` : ''; + return `${date} ${time}${keepFrac}`; +} + +function tree(value, key) { + if (value === null || typeof value !== 'object') { + const row = h('div', 'jv'); + if (key !== undefined) row.append(h('span', 'jk', `${key}: `)); + const text = typeof value === 'string' ? (formatTimestamp(value) ?? JSON.stringify(value)) : JSON.stringify(value); + row.append(h('span', `jval ${value === null ? 'jnull' : typeof value}`, text)); + return row; + } + const d = h('details', 'jt'); + d.open = true; + const isArr = Array.isArray(value); + const n = isArr ? value.length : Object.keys(value).length; + d.append(h('summary', null, `${key !== undefined ? `${key}: ` : ''}${isArr ? `[ ${n} ]` : `{ ${n} }`}`)); + for (const [k, v] of Object.entries(value)) d.append(tree(v, isArr ? Number(k) : k)); + return d; +} + +const meta = (r, extra = '') => h('div', 'rmeta', `${extra}${extra ? ' · ' : ''}${r.elapsedMs ?? 0} ms${r.truncated ? ' · truncated (500 rows / 1 MB limit)' : ''}`); + +export function renderResult(r) { + const box = h('div', `result ${r.kind}`); + switch (r.kind) { + case 'rows': { + const t = h('table', 'grid'); + const tr = h('tr'); + r.columns.forEach((c) => tr.append(h('th', null, c))); + const thead = h('thead'); + thead.append(tr); + t.append(thead); + const tb = h('tbody'); + for (const row of r.rows) { + const x = h('tr'); + for (const c of r.columns) { + const v = row[c]; + const td = h('td'); + if (v !== null && typeof v === 'object') td.append(tree(v)); + else if (v === null) td.textContent = 'NULL'; + else td.textContent = typeof v === 'string' ? (formatTimestamp(v) ?? v) : String(v); + x.append(td); + } + tb.append(x); + } + t.append(tb); + box.append(t, meta(r, `${r.rowCount} row${r.rowCount === 1 ? '' : 's'}`)); + break; + } + case 'docs': + r.docs.forEach((d) => box.append(tree(d))); + box.append(meta(r, `${r.count} document${r.count === 1 ? '' : 's'}`)); + break; + case 'dml': box.append(h('div', 'msg', `${r.rowsAffected} rows affected`), meta(r)); break; + case 'ok': box.append(h('div', 'msg', 'Statement executed'), meta(r)); break; + case 'count': box.append(h('div', 'msg', `count: ${r.count}`), meta(r)); break; + case 'collections': box.append(h('div', 'msg', r.names.join(' · ') || '(no collections)'), meta(r)); break; + case 'write': box.append(tree(r.result), meta(r)); break; + case 'error': box.append(h('div', 'msg', r.error), meta(r, r.code ?? 'error')); box.classList.add('error'); break; + default: box.append(h('div', 'msg', JSON.stringify(r))); + } + return box; +} + +export function renderRun(run) { + const wrap = h('div', 'run'); + if (run.cached) wrap.append(h('span', 'badge cool', 'served from cache')); + run.results.forEach((r) => wrap.append(renderResult(r))); + return wrap; +} diff --git a/labs/converged-modeling-patterns/app/scripts/calibrate-measure.mjs b/labs/converged-modeling-patterns/app/scripts/calibrate-measure.mjs new file mode 100644 index 0000000..17b320d --- /dev/null +++ b/labs/converged-modeling-patterns/app/scripts/calibrate-measure.mjs @@ -0,0 +1,91 @@ +// Measures document-model redo ÷ converged redo for each pattern's @measure pair at several +// sizes of its size knob, in a scratch schema on the lab database (never CMP_USER or WS_*). +// Protocol = the console's Measure it: in-memory undo off, one unmeasured warm-up, then 3 +// measured runs (median), each rolled back. Prints YAML to paste into each README's front matter. +// +// Usage: DB_HOST=localhost DB_PORT=1522 ORACLE_PASSWORD=... node scripts/calibrate-measure.mjs [patternId ...] +import fs from 'node:fs'; +import path from 'node:path'; +import oracledb from 'oracledb'; +import { loadPatterns } from '../src/content/patterns.js'; +import { parseSqlFile } from '../src/content/sqlParser.js'; +import { readStats } from '../src/db/oracle.js'; + +const ROOT = path.resolve(import.meta.dirname, '../../patterns'); +const SIZES = { + '01-extended-reference': [1, 10, 100, 1000], + '02-computed': [10, 100, 1000, 5000], + '03-bucket': [3, 100, 1000, 3600], + '04-subset': [3, 10, 100, 1000], + '05-tree-hierarchy': [3, 30, 300, 3000], + '06-outlier': [10, 100, 800, 2000], +}; +const USER = 'ZZ_CALIBRATE'; +const PW = 'Calibrate2026x'; +const host = process.env.DB_HOST ?? 'localhost'; +const port = process.env.DB_PORT ?? '1522'; +const connectString = `${host}:${port}/FREEPDB1`; +const median = (a) => [...a].sort((x, y) => x - y)[Math.floor(a.length / 2)]; + +async function asSystem(fn) { + const c = await oracledb.getConnection({ user: 'system', password: process.env.ORACLE_PASSWORD, connectString }); + try { return await fn(c); } finally { await c.close(); } +} + +async function run(conn, stmts, binds = {}) { + for (const s of stmts) { + const b = s.sql.includes(':n') ? binds : {}; + await conn.execute(s.sql, b); + } +} + +async function measureOnce(conn, sql) { + const before = await readStats(conn); + await conn.execute(sql); + const after = await readStats(conn); + await conn.rollback(); + return after['redo size'] - before['redo size']; +} + +async function calibrate(p) { + const conn = await oracledb.getConnection({ user: USER, password: PW, connectString }); + try { + await conn.execute('ALTER SESSION SET "_in_memory_undo" = false'); + await run(conn, p.setup.document); + await run(conn, p.setup.converged); + await conn.commit(); + const scale = parseSqlFile(fs.readFileSync(path.join(ROOT, p.id, 'calibrate.sql'), 'utf8')); + const m = p.measures[0]; + const points = []; + for (const n of SIZES[p.id]) { + await run(conn, scale, { n }); + await conn.commit(); + await measureOnce(conn, m.document.sql); + await measureOnce(conn, m.converged.sql); + const d = []; const c = []; + for (let i = 0; i < 3; i++) { d.push(await measureOnce(conn, m.document.sql)); c.push(await measureOnce(conn, m.converged.sql)); } + const ratio = median(d) / median(c); + points.push({ x: n, ratio: Number(ratio.toFixed(2)), doc: median(d), conv: median(c) }); + console.error(`${p.id} n=${n}: doc ${median(d)} B · conv ${median(c)} B · ${ratio.toFixed(1)}×`); + } + return points; + } finally { await conn.close(); } +} + +const only = process.argv.slice(2); +const patterns = loadPatterns(ROOT).filter((p) => !only.length || only.includes(p.id)); +await asSystem(async (c) => { + await c.execute(`BEGIN EXECUTE IMMEDIATE 'DROP USER ${USER} CASCADE'; EXCEPTION WHEN OTHERS THEN IF SQLCODE != -1918 THEN RAISE; END IF; END;`); + await c.execute(`CREATE USER ${USER} IDENTIFIED BY "${PW}" QUOTA UNLIMITED ON users`); + // Same object privileges a lab workspace gets (03 builds a materialized view, 05 a property graph). + await c.execute(`GRANT CREATE SESSION, CREATE TABLE, CREATE VIEW, CREATE TRIGGER, CREATE SEQUENCE, CREATE PROCEDURE, CREATE MATERIALIZED VIEW, CREATE PROPERTY GRAPH, SELECT ANY DICTIONARY TO ${USER}`); + await c.execute(`GRANT ALTER SESSION TO ${USER}`); +}); +try { + for (const p of patterns) { + const pts = await calibrate(p); + console.log(`# ${p.id}\nmeasure:\n calibration:\n${pts.map((q) => ` - { x: ${q.x}, ratio: ${q.ratio} } # doc ${q.doc} B, conv ${q.conv} B`).join('\n')}\n`); + } +} finally { + await asSystem((c) => c.execute(`DROP USER ${USER} CASCADE`)); +} diff --git a/labs/converged-modeling-patterns/app/scripts/help-review.mjs b/labs/converged-modeling-patterns/app/scripts/help-review.mjs new file mode 100644 index 0000000..66ed512 --- /dev/null +++ b/labs/converged-modeling-patterns/app/scripts/help-review.mjs @@ -0,0 +1,31 @@ +// Generates docs/help-review.md: every help entry by pattern, surface and statement, so the +// author can review all lab guidance text in one place. Flags cards that have no help. +import fs from 'node:fs'; +import path from 'node:path'; +import { loadPatterns } from '../src/content/patterns.js'; + +const root = path.resolve(import.meta.dirname, '../../patterns'); +const out = []; +const line = (s = '') => out.push(s); +const help = (hp) => [hp?.why && `- **Why:** ${hp.why}`, hp?.look && `- **Look for:** ${hp.look}`, hp?.figure && `- **Figure:** \`${hp.figure.file}\`: ${hp.figure.caption ?? ''}`].filter(Boolean); +let missing = 0; +line('# Lab help text: review'); line(); line('Generated by `app/scripts/help-review.mjs`. Edit the pattern files, not this file.'); line(); +for (const p of loadPatterns(root)) { + line(`## ${p.id} · ${p.meta.title}`); line(); + line('### Knobs'); for (const k of p.meta.knobs) line(`- **${k.name}** (${k.setting}): ${k.help ?? '⚠ MISSING'}`); line(); + // The MongoDB API tab only renders when the pattern has mongosh cards, so only then does it need help. + const tabKeys = ['document', 'converged', ...(p.lanes.mongo.length ? ['mongo'] : []), 'measure']; + line('### Tabs'); for (const k of tabKeys) { const t = p.meta.help.tabs[k]; line(`**${k}**`); const hl = help(t); if (!hl.length) { line('- ⚠ MISSING'); missing++; } else hl.forEach(line); line(); } + for (const [lane, steps] of Object.entries(p.lanes)) { + if (!steps.length) continue; + line(`### ${lane} cards`); line(); + for (const st of steps) { + line(`**${st.title}**${st.measure ? ` · measured: ${st.measure}` : ''}`); + const hl = help(st.help); if (!hl.length) { line('- ⚠ MISSING'); missing++; } else hl.forEach(line); + line(); + } + } +} +out.splice(3, 0, `Entries missing help: **${missing}**`, ''); +fs.writeFileSync(path.resolve(import.meta.dirname, '../../docs/help-review.md'), out.join('\n')); +console.log(`docs/help-review.md written · ${missing} missing`); diff --git a/labs/converged-modeling-patterns/app/sql/bootstrap.sql b/labs/converged-modeling-patterns/app/sql/bootstrap.sql new file mode 100644 index 0000000..2ccf5de --- /dev/null +++ b/labs/converged-modeling-patterns/app/sql/bootstrap.sql @@ -0,0 +1,37 @@ +-- app/sql/bootstrap.sql +-- Idempotent. Run once at lab-ui start-up as SYS AS SYSDBA connected to FREEPDB1. +-- Creates LAB_ADMIN (least privilege for provisioning attendee workspaces), the control +-- tables, and the V$ grants measure-it needs. :pw is bound to LAB_ADMIN_PASSWORD. +DECLARE + PROCEDURE x(p_sql VARCHAR2, p_ok1 PLS_INTEGER DEFAULT 0, p_ok2 PLS_INTEGER DEFAULT 0) IS + BEGIN + EXECUTE IMMEDIATE p_sql; + EXCEPTION WHEN OTHERS THEN + IF SQLCODE NOT IN (p_ok1, p_ok2) THEN RAISE; END IF; + END; +BEGIN + x('CREATE USER lab_admin IDENTIFIED BY "' || :pw || '" QUOTA 100M ON users', -1920); + x('ALTER USER lab_admin IDENTIFIED BY "' || :pw || '"'); + x('GRANT CREATE SESSION, CREATE TABLE TO lab_admin'); + x('GRANT CREATE USER, ALTER USER, DROP USER TO lab_admin'); + x('GRANT CREATE SESSION, CREATE TABLE, CREATE VIEW, CREATE PROCEDURE, CREATE TRIGGER, CREATE SEQUENCE, ' + || 'CREATE MATERIALIZED VIEW, CREATE PROPERTY GRAPH TO lab_admin WITH ADMIN OPTION'); + x('GRANT SODA_APP TO lab_admin WITH ADMIN OPTION'); + x('GRANT SELECT ON sys.v_$mystat TO lab_admin WITH GRANT OPTION'); + x('GRANT SELECT ON sys.v_$statname TO lab_admin WITH GRANT OPTION'); + x('GRANT SELECT ON sys.v_$session TO lab_admin'); + x('GRANT ALTER SYSTEM TO lab_admin'); -- reaper: ALTER SYSTEM KILL SESSION on a parked workspace's stuck sessions + x('GRANT SELECT ON sys.v_$mystat TO cmp_user'); + x('GRANT SELECT ON sys.v_$statname TO cmp_user'); + x('GRANT ORDS_ADMINISTRATOR_ROLE TO lab_admin', -1919); -- role exists once ORDS is installed + x('CREATE TABLE lab_admin.lab_users (schema_name VARCHAR2(30) PRIMARY KEY, schema_password VARCHAR2(64) NOT NULL, ' + || 'email VARCHAR2(320) UNIQUE, display_name VARCHAR2(200), created_at TIMESTAMP DEFAULT SYSTIMESTAMP, assigned_at TIMESTAMP)', -955); + x('CREATE TABLE lab_admin.lab_dirty (schema_name VARCHAR2(30), pattern_id VARCHAR2(64), ' + || 'CONSTRAINT lab_dirty_pk PRIMARY KEY (schema_name, pattern_id))', -955); + x('CREATE TABLE lab_admin.lab_built (schema_name VARCHAR2(30), pattern_id VARCHAR2(64), version VARCHAR2(16) NOT NULL, ' + || 'CONSTRAINT lab_built_pk PRIMARY KEY (schema_name, pattern_id))', -955); + x('CREATE TABLE lab_admin.lab_settings (k VARCHAR2(64) PRIMARY KEY, v VARCHAR2(4000))', -955); + x('CREATE TABLE lab_admin.lab_pending_drop (schema_name VARCHAR2(30) PRIMARY KEY, ' + || 'requested_at TIMESTAMP DEFAULT SYSTIMESTAMP, last_error VARCHAR2(400))', -955); +END; +/ diff --git a/labs/converged-modeling-patterns/app/src/cache.js b/labs/converged-modeling-patterns/app/src/cache.js new file mode 100644 index 0000000..29c08d7 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/cache.js @@ -0,0 +1,92 @@ +import crypto from 'node:crypto'; + +export function normalizeSql(text) { + return String(text) + .replace(/\/\*[\s\S]*?\*\//g, ' ') + .replace(/--[^\n]*/g, ' ') + .replace(/\s+/g, ' ') + .trim() + .replace(/;\s*$/, '') + .trim(); +} + +export function isReadOnlySql(statements) { + if (statements.length !== 1) return false; + const n = normalizeSql(statements[0]); + return /^(SELECT|WITH)\b/i.test(n) && !/\bFOR\s+UPDATE\b/i.test(n); +} + +const MONGO_READ_OPS = new Set(['find', 'findOne', 'countDocuments']); + +// The SQL text of a $sql stage: the string form, or the object form's `statement`. +// null when the object form carries no string statement (the caller treats that as a write). +export function sqlStageText(value) { + if (typeof value === 'string') return value; + if (value && typeof value === 'object' && typeof value.statement === 'string') return value.statement; + return null; +} + +export function isReadOnlyMongo(cmd) { + if (cmd.kind === 'showCollections') return true; + if (MONGO_READ_OPS.has(cmd.op)) return true; + if (cmd.op !== 'aggregate') return false; + const pipeline = cmd.args[0] ?? []; + return pipeline.every((stage) => { + if ('$out' in stage || '$merge' in stage) return false; + if (!('$sql' in stage)) return true; + const sql = sqlStageText(stage.$sql); + return sql !== null && isReadOnlySql([sql]); + }); +} + +export const PATTERN_PREFIXES = { + xr: '01-extended-reference', cp: '02-computed', bk: '03-bucket', sb: '04-subset', tr: '05-tree-hierarchy', ol: '06-outlier', +}; + +export function patternsTouched(text, allIds) { + const found = new Set(); + for (const m of String(text).matchAll(/\b(xr|cp|bk|sb|tr|ol)_\w+/gi)) { + const id = PATTERN_PREFIXES[m[1].toLowerCase()]; + if (allIds.includes(id)) found.add(id); + } + return found.size ? [...found] : [...allIds]; +} + +export class ResultCache { + #map = new Map(); + #max; + enabled = true; + + constructor({ maxEntries = 500 } = {}) { + this.#max = maxEntries; + } + + // Only card text is ever cached (the runner checks), so the key is the raw text with + // leading/trailing whitespace trimmed — no comment or whitespace normalization inside. + key(patternId, text, version) { + const h = crypto.createHash('sha256').update(String(text).trim()).digest('hex'); + return `${patternId}|${version}|${h}`; + } + + get(key) { + if (!this.enabled || !this.#map.has(key)) return undefined; + const v = this.#map.get(key); + this.#map.delete(key); + this.#map.set(key, v); + return v; + } + + set(key, value) { + this.#map.delete(key); + this.#map.set(key, value); + while (this.#map.size > this.#max) this.#map.delete(this.#map.keys().next().value); + } + + clear() { + this.#map.clear(); + } + + get size() { + return this.#map.size; + } +} diff --git a/labs/converged-modeling-patterns/app/src/config.js b/labs/converged-modeling-patterns/app/src/config.js new file mode 100644 index 0000000..fe046af --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/config.js @@ -0,0 +1,62 @@ +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const here = path.dirname(fileURLToPath(import.meta.url)); + +function int(env, key, dflt, { min = 1 } = {}) { + if (env[key] === undefined || env[key] === '') return dflt; + const n = Number(env[key]); + if (!Number.isInteger(n) || n < min) throw new Error(`${key} must be an integer >= ${min} (got "${env[key]}")`); + return n; +} + +function deepFreeze(o) { + Object.values(o).forEach((v) => v && typeof v === 'object' && deepFreeze(v)); + return Object.freeze(o); +} + +// The repo's published default for LAB_ADMIN (compose.yml). Fine for a one-person solo lab; +// in event mode the room can reach the console, so it must be changed. +const DEFAULT_LAB_ADMIN_PASSWORD = 'LabAdmin2026'; + +// `allowDefaultSecrets` exists for the integration harness only: it exercises event-mode +// code paths against the lab's own database, whose LAB_ADMIN password is the default. +export function loadConfig(env = process.env, { allowDefaultSecrets = false } = {}) { + const mode = env.LAB_MODE || 'solo'; + if (!['solo', 'event'].includes(mode)) throw new Error(`LAB_MODE must be "solo" or "event" (got "${mode}")`); + // Bootstrap DDL embeds LAB_ADMIN_PASSWORD in a quoted identifier; a " would break it. + for (const key of ['ORACLE_PASSWORD', 'CMP_PASSWORD', 'LAB_ADMIN_PASSWORD']) { + if (String(env[key] ?? '').includes('"')) throw new Error(`${key} must not contain " (double quote)`); + } + const adminPassword = env.ADMIN_PASSWORD || null; + const labAdminPassword = env.LAB_ADMIN_PASSWORD || DEFAULT_LAB_ADMIN_PASSWORD; + if (mode === 'event' && !adminPassword) throw new Error('ADMIN_PASSWORD is required when LAB_MODE=event'); + if (mode === 'event' && !allowDefaultSecrets && labAdminPassword === DEFAULT_LAB_ADMIN_PASSWORD) { + throw new Error('LAB_ADMIN_PASSWORD is still the repo default (LabAdmin2026); set your own before starting LAB_MODE=event'); + } + return deepFreeze({ + mode, + port: int(env, 'PORT', 3000), + patternsDir: env.PATTERNS_DIR || path.resolve(here, '../../patterns'), + db: { + host: env.DB_HOST || 'oracle', + port: int(env, 'DB_PORT', 1521), + service: env.DB_SERVICE || 'FREEPDB1', + sysPassword: env.ORACLE_PASSWORD || 'Sandbox2026', + cmpUser: 'CMP_USER', + cmpPassword: env.CMP_PASSWORD || 'CmpUser2026', + labAdminPassword, + poolMax: int(env, 'DB_POOL_MAX', 1), + }, + mongo: { host: env.MONGO_HOST || env.DB_HOST || 'oracle', port: int(env, 'MONGO_PORT', 27017), poolMax: int(env, 'MONGO_POOL_MAX', 1) }, + gate: { + permits: 1, + queueTimeoutMs: int(env, 'QUEUE_TIMEOUT_MS', 30000), + sqlTimeoutMs: int(env, 'SQL_TIMEOUT_MS', 10000), + mongoTimeoutMs: int(env, 'MONGO_TIMEOUT_MS', 10000), + }, + cache: { maxEntries: int(env, 'CACHE_MAX_ENTRIES', 500), enabled: env.CACHE_ENABLED !== 'false' }, + limits: { maxRows: 500, maxBytes: 1048576, maxStatements: 20 }, + event: { code: env.EVENT_CODE || null, adminPassword }, + }); +} diff --git a/labs/converged-modeling-patterns/app/src/content/mongoCommand.js b/labs/converged-modeling-patterns/app/src/content/mongoCommand.js new file mode 100644 index 0000000..570e1c1 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/content/mongoCommand.js @@ -0,0 +1,94 @@ +import * as acorn from 'acorn'; +import { ObjectId } from 'mongodb'; + +export class MongoParseError extends Error {} + +const COLL_OPS = new Set(['find', 'findOne', 'aggregate', 'countDocuments', 'insertOne', 'insertMany', + 'updateOne', 'updateMany', 'deleteOne', 'deleteMany']); +const CURSOR_MODS = new Set(['sort', 'limit', 'skip', 'toArray']); + +const fail = (node, msg) => new MongoParseError(node && node.start !== undefined ? `${msg} (at char ${node.start})` : msg); + +function lit(node) { + switch (node.type) { + case 'Literal': + return node.regex ? new RegExp(node.regex.pattern, node.regex.flags) : node.value; + case 'TemplateLiteral': + if (node.expressions.length) throw fail(node, 'template literals with ${…} are not supported'); + return node.quasis.map((q) => q.value.cooked).join(''); + case 'ArrayExpression': + return node.elements.map((el) => { + if (!el || el.type === 'SpreadElement') throw fail(node, 'unsupported array element'); + return lit(el); + }); + case 'ObjectExpression': { + const o = {}; + for (const p of node.properties) { + if (p.type !== 'Property' || p.computed || p.kind !== 'init' || p.method) throw fail(p, 'unsupported object property'); + o[p.key.type === 'Identifier' ? p.key.name : String(p.key.value)] = lit(p.value); + } + return o; + } + case 'UnaryExpression': + if (node.operator === '-' && node.argument.type === 'Literal' && typeof node.argument.value === 'number') return -node.argument.value; + break; + case 'NewExpression': + case 'CallExpression': { + const name = node.callee.type === 'Identifier' ? node.callee.name : null; + if (name === 'Date' || name === 'ISODate') { + const a = node.arguments.map(lit); + return a.length ? new Date(a[0]) : new Date(); + } + if (name === 'ObjectId') return new ObjectId(lit(node.arguments[0])); + break; + } + default: + break; + } + throw fail(node, `unsupported syntax: ${node.type}`); +} + +const prop = (m) => (m.computed ? (m.property.type === 'Literal' ? String(m.property.value) : null) : m.property.name); +const isDb = (n) => n.type === 'Identifier' && n.name === 'db'; + +function collectionName(n) { + if (n.type === 'MemberExpression' && isDb(n.object)) return prop(n); + if (n.type === 'CallExpression' && n.callee.type === 'MemberExpression' && isDb(n.callee.object) && prop(n.callee) === 'getCollection') { + return lit(n.arguments[0]); + } + return null; +} + +export function parseMongoCommand(text) { + const src = String(text).trim().replace(/;\s*$/, ''); + if (/^show\s+collections$/i.test(src)) return { kind: 'showCollections' }; + let expr; + try { + expr = acorn.parseExpressionAt(src, 0, { ecmaVersion: 'latest' }); + } catch (e) { + throw new MongoParseError(`could not parse: ${e.message}`); + } + if (src.slice(expr.end).trim()) throw new MongoParseError('only one command at a time'); + const shapeErr = () => new MongoParseError('expected db..(…) or db.aggregate([…])'); + + const mods = {}; + let node = expr; + while (node.type === 'CallExpression' && node.callee.type === 'MemberExpression' && CURSOR_MODS.has(prop(node.callee))) { + const name = prop(node.callee); + if (name !== 'toArray') mods[name] = lit(node.arguments[0]); + node = node.callee.object; + } + if (node.type !== 'CallExpression' || node.callee.type !== 'MemberExpression') throw shapeErr(); + const op = prop(node.callee); + const target = node.callee.object; + const args = node.arguments.map(lit); + if (isDb(target)) { + if (op !== 'aggregate') throw new MongoParseError(`unsupported operation db.${op}`); + return { kind: 'db', op, args, mods }; + } + const collection = collectionName(target); + if (!collection) throw shapeErr(); + if (!COLL_OPS.has(op)) throw new MongoParseError(`unsupported operation ${op}`); + if (Object.keys(mods).length && op !== 'find') throw new MongoParseError('sort/limit/skip only apply to find'); + return { kind: 'collection', collection, op, args, mods }; +} diff --git a/labs/converged-modeling-patterns/app/src/content/mongoScript.js b/labs/converged-modeling-patterns/app/src/content/mongoScript.js new file mode 100644 index 0000000..ca6ea02 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/content/mongoScript.js @@ -0,0 +1,50 @@ +import * as acorn from 'acorn'; +import { parseMongoCommand } from './mongoCommand.js'; +import { addHelp } from './sqlParser.js'; + +const ANNOT = /^\s*@(step|note|why|look|figure)\b\s*(.*)$/i; + +export function parseMongoScript(text, file = 'script') { + const comments = []; + const ast = acorn.parse(text, { ecmaVersion: 'latest', sourceType: 'script', locations: true, onComment: comments }); + const annos = comments + .filter((c) => c.type === 'Line') + .map((c) => ({ c, m: c.value.match(ANNOT) })) + .filter((x) => x.m); + const steps = []; + let ai = 0; + let prevEnd = 0; + for (const st of ast.body) { + const mine = []; + while (ai < annos.length && annos[ai].c.end <= st.start) { + if (annos[ai].c.start >= prevEnd) mine.push(annos[ai]); + ai++; + } + prevEnd = st.end; + const stepAnno = mine.filter((x) => x.m[1].toLowerCase() === 'step').pop(); + if (!stepAnno) continue; + const node = st.type === 'VariableDeclaration' && st.declarations.length === 1 ? st.declarations[0].init + : st.type === 'ExpressionStatement' ? st.expression : null; + const where = `${file}:${st.loc.start.line}`; + if (!node) throw new Error(`${where}: @step must precede a db.* expression or "const x = db.*"`); + const command = text.slice(node.start, node.end).replace(/\.toArray\(\s*\)\s*$/, ''); + try { + parseMongoCommand(command); + } catch (e) { + throw new Error(`${where}: ${e.message}`); + } + const help = { why: null, look: null, figure: null }; + for (const x of mine) { + const k = x.m[1].toLowerCase(); + if (k === 'why' || k === 'look' || k === 'figure') addHelp(help, k, x.m[2].trim(), where); + } + steps.push({ + title: stepAnno.m[2].trim(), + notes: mine.filter((x) => x.m[1].toLowerCase() === 'note').map((x) => x.m[2].trim()), + help, + command, + line: st.loc.start.line, + }); + } + return steps; +} diff --git a/labs/converged-modeling-patterns/app/src/content/patterns.js b/labs/converged-modeling-patterns/app/src/content/patterns.js new file mode 100644 index 0000000..64c7b12 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/content/patterns.js @@ -0,0 +1,98 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import crypto from 'node:crypto'; +import YAML from 'yaml'; +import { parseSqlFile, isSetup } from './sqlParser.js'; +import { parseMongoScript } from './mongoScript.js'; + +function frontMatter(file) { + const text = fs.readFileSync(file, 'utf8').replace(/\r\n/g, '\n'); + const m = text.match(/^---\n([\s\S]*?)\n---\n/); + if (!m) throw new Error(`${file}: missing YAML front matter block`); + const fm = YAML.parse(m[1]); + for (const k of ['title', 'industry', 'deck', 'problem']) { + if (!fm?.[k]) throw new Error(`${file}: front matter needs "${k}"`); + } + if (!Array.isArray(fm.knobs) || fm.knobs.length !== 3) throw new Error(`${file}: front matter needs exactly 3 knobs`); + const FIG = /^[a-z0-9-]+\.svg$/; + const tabHelp = (t, k) => { + if (t == null) return undefined; + if (t.figure != null && !FIG.test(String(t.figure))) throw new Error(`${file}: help.tabs.${k}.figure must be a plain .svg name`); + return { why: t.why ? String(t.why) : null, look: t.look ? String(t.look) : null, + figure: t.figure ? { file: String(t.figure), caption: t.caption ? String(t.caption) : null } : null }; + }; + const tabs = {}; + for (const k of ['document', 'converged', 'mongo', 'measure']) { + const v = tabHelp(fm.help?.tabs?.[k], k); + if (v) tabs[k] = v; + } + let measure = null; + if (fm.measure) { + const cal = (fm.measure.calibration ?? []).map((c) => ({ x: Number(c.x), ratio: Number(c.ratio) })); + if (cal.length < 4 || cal.some((c) => !(c.x > 0) || !(c.ratio > 0))) throw new Error(`${file}: measure.calibration needs ≥ 4 points with x > 0 and ratio > 0`); + cal.sort((a, b) => a.x - b.x); + if (typeof fm.measure.x_label !== 'string' || !fm.measure.x_label.trim()) throw new Error(`${file}: measure.x_label must be a non-empty string`); + const labX = Number(fm.measure.lab_x); + if (!(labX >= cal[0].x && labX <= cal[cal.length - 1].x)) throw new Error(`${file}: measure.lab_x must lie inside the calibrated x range`); + measure = { xLabel: String(fm.measure.x_label), labX, deckSlides: String(fm.measure.deck_slides ?? fm.deck), calibration: cal }; + } + return { + title: String(fm.title), + industry: String(fm.industry), + deck: String(fm.deck), + problem: String(fm.problem).trim(), + knobs: fm.knobs.map((k) => ({ name: String(k.name), setting: String(k.setting), hot: k.hot === true, help: k.help ? String(k.help) : null })), + help: { tabs }, + measure, + }; +} + +const readSql = (p) => (fs.existsSync(p) ? parseSqlFile(fs.readFileSync(p, 'utf8')) : []); + +function checkMeasure(stmt, where) { + const ok = stmt.plsql || /^(UPDATE|INSERT|DELETE|MERGE)\b/i.test(stmt.sql); + if (!ok) throw new Error(`${where}: measure "${stmt.measure}" must be UPDATE/INSERT/DELETE/MERGE or PL/SQL`); + if (/\bCOMMIT\b/i.test(stmt.sql)) throw new Error(`${where}: measure "${stmt.measure}" must not COMMIT (it runs in a rolled-back transaction)`); +} + +function loadOne(root, id) { + const dir = path.join(root, id); + const meta = frontMatter(path.join(dir, 'README.md')); + const docFile = path.join(dir, '01-document-model.sql'); + const convFile = path.join(dir, '02-converged.sql'); + const doc = readSql(docFile); + const conv = readSql(convFile); + const mongo = fs.readdirSync(dir) + .filter((n) => n.endsWith('.js')) + .sort() + .flatMap((n) => parseMongoScript(fs.readFileSync(path.join(dir, n), 'utf8'), path.join(dir, n))); + + const measures = []; + const docTags = new Map(doc.filter((s) => s.measure).map((s) => [s.measure, s])); + const convTags = new Map(conv.filter((s) => s.measure).map((s) => [s.measure, s])); + for (const tag of new Set([...docTags.keys(), ...convTags.keys()])) { + if (!docTags.has(tag) || !convTags.has(tag)) throw new Error(`${dir}: measure "${tag}" must appear in both 01-document-model.sql and 02-converged.sql`); + checkMeasure(docTags.get(tag), docFile); + checkMeasure(convTags.get(tag), convFile); + measures.push({ tag, document: docTags.get(tag), converged: convTags.get(tag) }); + } + + const lanes = { document: doc.filter((s) => s.title), converged: conv.filter((s) => s.title), mongo }; + if (!lanes.document.length || !lanes.converged.length) throw new Error(`${dir}: needs at least one @step in each .sql file`); + if (!measures.length) throw new Error(`${dir}: needs at least one @measure pair`); + + const setup = { document: doc.filter(isSetup), converged: conv.filter(isSetup) }; + const version = crypto.createHash('sha256') + .update([...setup.document, ...setup.converged].map((s) => s.sql).join('\n;\n')) + .digest('hex') + .slice(0, 16); + return { id, meta, lanes, setup, measures, version }; +} + +export function loadPatterns(root) { + return fs.readdirSync(root, { withFileTypes: true }) + .filter((d) => d.isDirectory() && /^\d\d-/.test(d.name)) + .map((d) => d.name) + .sort() + .map((id) => loadOne(root, id)); +} diff --git a/labs/converged-modeling-patterns/app/src/content/sqlParser.js b/labs/converged-modeling-patterns/app/src/content/sqlParser.js new file mode 100644 index 0000000..939f5fc --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/content/sqlParser.js @@ -0,0 +1,112 @@ +// Parses the validated pattern .sql files (and console input) into executable statements. +// Limitation: q'[...]' literals and multi-line /* */ comments inside a statement are not +// supported; the pattern files use neither. +// SET TRANSACTION / SET ROLE are SQL statements, not SQL*Plus settings. +const SQLPLUS = /^(SET(?!\s+(TRANSACTION|ROLE)\b)|PROMPT|COLUMN|COL|WHENEVER|EXIT|QUIT|SPOOL|DEFINE|UNDEFINE|SHOW|TTITLE|BTITLE|BREAK|CLEAR)\b|^@/i; +const PLSQL_START = /^(BEGIN|DECLARE)\b|^CREATE\s+(OR\s+REPLACE\s+)?((NON)?EDITIONABLE\s+)?(TRIGGER|PROCEDURE|FUNCTION|PACKAGE|TYPE)\b/i; +const ANNOT = /^--\s*@(step|note|measure|why|look|figure)\b\s*(.*)$/i; +const FIGURE = /^([a-z0-9-]+\.svg)\s+(.+)$/; + +// Shared by the SQL and mongosh parsers: fold one help annotation into an accumulator. +export function addHelp(help, kind, val, where = 'annotation') { + if (kind === 'figure') { + const f = val.match(FIGURE); + if (!f) throw new Error(`${where}: @figure needs ".svg " with a plain file name`); + help.figure = { file: f[1], caption: f[2].trim() }; + } else { + help[kind] = help[kind] ? `${help[kind]} ${val}` : val; + } +} +const emptyHelp = () => ({ why: null, look: null, figure: null }); + +function scanLine(line, inString) { + for (let i = 0; i < line.length; i++) { + const ch = line[i]; + if (inString) { + if (ch === "'") { + if (line[i + 1] === "'") i++; + else inString = false; + } + continue; + } + if (ch === "'") { inString = true; continue; } + if (ch === '-' && line[i + 1] === '-') return { end: -1, inString }; + if (ch === '/' && line[i + 1] === '*') { + const close = line.indexOf('*/', i + 2); + if (close === -1) return { end: -1, inString }; + i = close + 1; + continue; + } + if (ch === ';') return { end: i, inString }; + } + return { end: -1, inString }; +} + +export function parseSqlFile(text) { + const lines = String(text).replace(/\r\n/g, '\n').split('\n'); + const out = []; + let anno = null; + let buf = []; + let startLine = 0; + let plsql = false; + let inString = false; + + const flush = () => { + const sql = buf.join('\n').trim(); + if (sql) { + out.push({ sql, title: anno?.title ?? null, notes: anno?.notes ?? [], measure: anno?.measure ?? null, + help: anno?.help ?? emptyHelp(), plsql, line: startLine }); + } + anno = null; + buf = []; + plsql = false; + inString = false; + }; + + const handle = (line, idx) => { + const t = line.trim(); + if (buf.length === 0) { + if (!t || t === '/') return; + const a = t.match(ANNOT); + if (a) { + anno ??= { title: null, notes: [], measure: null, help: emptyHelp() }; + const kind = a[1].toLowerCase(); + const val = a[2].trim(); + if (kind === 'step') anno.title = val; + else if (kind === 'note') anno.notes.push(val); + else if (kind === 'measure') anno.measure = val; + else addHelp(anno.help, kind, val, `line ${idx + 1}`); + return; + } + if (t.startsWith('--') || SQLPLUS.test(t)) return; + startLine = idx + 1; + plsql = PLSQL_START.test(t); + } + if (plsql) { + if (t === '/') flush(); + else buf.push(line); + return; + } + if (t === '/') { flush(); return; } + const r = scanLine(line, inString); + if (r.end >= 0) { + buf.push(line.slice(0, r.end)); + flush(); + // Keep scanning after the ';': "SELECT 1 FROM dual; SELECT 2 FROM dual;" is two statements. + const rest = line.slice(r.end + 1); + if (rest.trim()) handle(rest, idx); + return; + } + inString = r.inString; + buf.push(line); + }; + lines.forEach(handle); + if (buf.length) flush(); + return out; +} + +export const isSetup = (stmt) => stmt.title === null && stmt.measure === null; + +export function splitConsoleSql(text) { + return parseSqlFile(text).map((s) => s.sql); +} diff --git a/labs/converged-modeling-patterns/app/src/db/mongo.js b/labs/converged-modeling-patterns/app/src/db/mongo.js new file mode 100644 index 0000000..e760ddf --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/db/mongo.js @@ -0,0 +1,119 @@ +// app/src/db/mongo.js +import { performance } from 'node:perf_hooks'; +import { MongoClient, BSON } from 'mongodb'; + +const IDLE_MS = 5 * 60 * 1000; + +export function mongoUri({ host, port, user, password }) { + const u = encodeURIComponent(user); + const p = encodeURIComponent(password); + return `mongodb://${u}:${p}@${host}:${port}/${user.toUpperCase()}?authMechanism=PLAIN&authSource=$external&retryWrites=false&loadBalanced=true`; +} + +export class MongoPool { + #cfg; + #clients = new Map(); + + constructor(cfg) { + this.#cfg = cfg; + } + + client(user, password) { + const key = user.toUpperCase(); + let e = this.#clients.get(key); + if (!e) { + const c = new MongoClient(mongoUri({ host: this.#cfg.mongo.host, port: this.#cfg.mongo.port, user: key, password }), { + maxPoolSize: this.#cfg.mongo.poolMax, + }); + e = { client: c, timer: null }; + this.#clients.set(key, e); + } + clearTimeout(e.timer); + e.timer = setTimeout(() => this.drop(key), IDLE_MS); + e.timer.unref?.(); + return e.client; + } + + async drop(user) { + const key = user.toUpperCase(); + const e = this.#clients.get(key); + if (!e) return; + this.#clients.delete(key); + clearTimeout(e.timer); + await e.client.close(true).catch(() => {}); + } + + async closeAll() { + await Promise.all([...this.#clients.keys()].map((k) => this.drop(k))); + } +} + +const toJson = (doc) => BSON.EJSON.serialize(doc, { relaxed: true }); + +// The system timeout always wins: user-supplied options are spread first, then +// maxTimeMS is forced to timeoutMs last, so a caller can never raise the cap +// (e.g. `{ maxTimeMS: 600000 }` in a user aggregate/update options literal). +const withCap = (userOpts, timeoutMs) => ({ ...(userOpts ?? {}), maxTimeMS: timeoutMs }); + +async function take(cursor, maxRows, maxBytes) { + const docs = []; + let truncated = false; + for await (const d of cursor) { + if (docs.length === maxRows) { truncated = true; break; } + docs.push(toJson(d)); + } + await cursor.close(); + while (docs.length && JSON.stringify(docs).length > maxBytes) { + docs.splice(Math.floor(docs.length * 0.8)); + truncated = true; + } + return { docs, truncated }; +} + +export async function runMongo(client, dbName, cmd, { timeoutMs, maxRows, maxBytes }) { + const start = performance.now(); + const elapsedMs = () => Math.round(performance.now() - start); + const db = client.db(dbName); + const o = { maxTimeMS: timeoutMs }; + try { + if (cmd.kind === 'showCollections') { + const names = (await db.listCollections({}, { nameOnly: true, maxTimeMS: timeoutMs }).toArray()).map((c) => c.name).sort(); + return { kind: 'collections', names, elapsedMs: elapsedMs() }; + } + if (cmd.kind === 'db') { + const { docs, truncated } = await take(db.aggregate(cmd.args[0] ?? [], withCap(cmd.args[1], timeoutMs)), maxRows, maxBytes); + return { kind: 'docs', docs, count: docs.length, truncated, elapsedMs: elapsedMs() }; + } + const coll = db.collection(cmd.collection); + const [a0 = {}, a1, a2] = cmd.args; + switch (cmd.op) { + case 'find': { + let cur = coll.find(a0, { ...o, ...(a1 ? { projection: a1 } : {}) }); + if (cmd.mods.sort) cur = cur.sort(cmd.mods.sort); + if (cmd.mods.skip) cur = cur.skip(cmd.mods.skip); + cur = cur.limit(Math.min(cmd.mods.limit ?? maxRows + 1, maxRows + 1)); + const { docs, truncated } = await take(cur, maxRows, maxBytes); + return { kind: 'docs', docs, count: docs.length, truncated, elapsedMs: elapsedMs() }; + } + case 'findOne': { + const d = await coll.findOne(a0, { ...o, ...(a1 ? { projection: a1 } : {}) }); + return { kind: 'docs', docs: d ? [toJson(d)] : [], count: d ? 1 : 0, truncated: false, elapsedMs: elapsedMs() }; + } + case 'aggregate': { + const { docs, truncated } = await take(coll.aggregate(a0, withCap(a1, timeoutMs)), maxRows, maxBytes); + return { kind: 'docs', docs, count: docs.length, truncated, elapsedMs: elapsedMs() }; + } + case 'countDocuments': + return { kind: 'count', count: await coll.countDocuments(a0, o), elapsedMs: elapsedMs() }; + case 'insertOne': return { kind: 'write', result: toJson(await coll.insertOne(a0, o)), elapsedMs: elapsedMs() }; + case 'insertMany': return { kind: 'write', result: toJson(await coll.insertMany(a0, o)), elapsedMs: elapsedMs() }; + case 'updateOne': return { kind: 'write', result: toJson(await coll.updateOne(a0, a1, withCap(a2, timeoutMs))), elapsedMs: elapsedMs() }; + case 'updateMany': return { kind: 'write', result: toJson(await coll.updateMany(a0, a1, withCap(a2, timeoutMs))), elapsedMs: elapsedMs() }; + case 'deleteOne': return { kind: 'write', result: toJson(await coll.deleteOne(a0, o)), elapsedMs: elapsedMs() }; + case 'deleteMany': return { kind: 'write', result: toJson(await coll.deleteMany(a0, o)), elapsedMs: elapsedMs() }; + default: return { kind: 'error', error: `unsupported operation ${cmd.op}`, code: null, elapsedMs: elapsedMs() }; + } + } catch (err) { + return { kind: 'error', error: err.message, code: err.code ?? err.codeName ?? null, elapsedMs: elapsedMs() }; + } +} diff --git a/labs/converged-modeling-patterns/app/src/db/oracle.js b/labs/converged-modeling-patterns/app/src/db/oracle.js new file mode 100644 index 0000000..55e2316 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/db/oracle.js @@ -0,0 +1,164 @@ +// app/src/db/oracle.js +import fs from 'node:fs'; +import path from 'node:path'; +import { performance } from 'node:perf_hooks'; +import oracledb from 'oracledb'; +import { parseSqlFile } from '../content/sqlParser.js'; + +oracledb.fetchAsString = [oracledb.CLOB]; + +const BOOTSTRAP = path.resolve(import.meta.dirname, '../../sql/bootstrap.sql'); +export const STAT_NAMES = ['redo size', 'db block changes', 'session logical reads', 'CPU used by this session']; + +const connectString = (cfg) => `${cfg.db.host}:${cfg.db.port}/${cfg.db.service}`; + +export async function bootstrap(cfg) { + const conn = await oracledb.getConnection({ + user: 'sys', password: cfg.db.sysPassword, connectString: connectString(cfg), privilege: oracledb.SYSDBA, + }); + try { + for (const st of parseSqlFile(fs.readFileSync(BOOTSTRAP, 'utf8'))) { + await conn.execute(st.sql, st.sql.includes(':pw') ? { pw: cfg.db.labAdminPassword } : {}); + } + } finally { + await conn.close(); + } +} + +// node-oracledb Thin mode does not implement heterogeneous connection pooling +// (NJS-089: "Heterogeneous Pooling is not supported by node-oracledb in Thin mode") — +// this is a permanent Thin-mode architecture limitation, not a version bug, and switching +// to Thick mode requires review (see task brief). Attendee schemas are created dynamically +// by lab_admin, so their usernames are not known up front and cannot each get their own +// homogeneous pool. Instead `exec` is a minimal pool-shaped wrapper: it hands out standalone +// Thin-mode connections under whatever credentials are supplied, bounded to `cfg.db.poolMax` +// concurrent connections with the same queue-timeout behavior a real pool would apply. +// +// `connect` is injectable (defaults to a real oracledb connection) so the wrapper's own +// acquire/release/close bookkeeping can be unit-tested with a stub, independent of the DB. +export function createExecPool(cfg, { connect = (opts) => oracledb.getConnection({ ...opts, connectString: connectString(cfg) }) } = {}) { + const max = cfg.db.poolMax; + const queueTimeoutMs = cfg.gate.queueTimeoutMs; + let open = 0; + let closed = false; + const waiters = []; + const live = new Set(); + + function release() { + open--; + const next = waiters.shift(); + if (next) { + clearTimeout(next.timer); + open++; + next.resolve(); + } + } + + function acquire() { + if (open < max) { + open++; + return Promise.resolve(); + } + return new Promise((resolve, reject) => { + const waiter = { + resolve, + reject, + timer: setTimeout(() => { + const i = waiters.indexOf(waiter); + if (i !== -1) waiters.splice(i, 1); + reject(new Error('NJS-040: connection request timed out')); + }, queueTimeoutMs), + }; + waiters.push(waiter); + }); + } + + return { + async getConnection(opts = {}) { + if (closed) throw new Error('exec pool is closed'); + await acquire(); + let conn; + try { + conn = await connect(opts); + } catch (err) { + release(); + throw err; + } + let released = false; + const releaseOnce = () => { + if (released) return; + released = true; + live.delete(conn); + release(); + }; + const rawClose = conn.close.bind(conn); + conn.close = async (...args) => { + try { + return await rawClose(...args); + } finally { + releaseOnce(); + } + }; + live.add(conn); + return conn; + }, + async close() { + closed = true; + for (const w of waiters.splice(0)) { + clearTimeout(w.timer); + w.reject(new Error('exec pool is closing')); + } + const toClose = [...live]; + live.clear(); + await Promise.all(toClose.map((c) => c.close({ drop: true }).catch(() => {}))); + }, + }; +} + +export async function createPools(cfg) { + const exec = createExecPool(cfg); + const control = await oracledb.createPool({ + user: 'lab_admin', password: cfg.db.labAdminPassword, connectString: connectString(cfg), + poolMin: 0, poolMax: 1, poolIncrement: 1, queueTimeout: 60000, + }); + return { exec, control, close: async () => { await exec.close(); await control.close(5); } }; +} + +function clip(rows, maxRows, maxBytes) { + let truncated = rows.length > maxRows; + let out = rows.slice(0, maxRows); + while (out.length && JSON.stringify(out).length > maxBytes) { + out = out.slice(0, Math.floor(out.length * 0.8)); + truncated = true; + } + return { rows: out, truncated }; +} + +export async function executeSql(conn, sql, { timeoutMs, maxRows, maxBytes, autoCommit = true }) { + const start = performance.now(); + const elapsed = () => Math.round(performance.now() - start); + conn.callTimeout = timeoutMs; + try { + const r = await conn.execute(sql, [], { outFormat: oracledb.OUT_FORMAT_OBJECT, autoCommit, maxRows: maxRows + 1 }); + if (r.metaData) { + const { rows, truncated } = clip(r.rows ?? [], maxRows, maxBytes); + return { kind: 'rows', columns: r.metaData.map((m) => m.name), rows, rowCount: rows.length, truncated, elapsedMs: elapsed() }; + } + if (typeof r.rowsAffected === 'number') return { kind: 'dml', rowsAffected: r.rowsAffected, elapsedMs: elapsed() }; + return { kind: 'ok', elapsedMs: elapsed() }; + } catch (err) { + return { kind: 'error', error: err.message, code: err.code ?? null, elapsedMs: elapsed() }; + } finally { + conn.callTimeout = 0; + } +} + +export async function readStats(conn) { + const r = await conn.execute( + `SELECT n.name, m.value FROM v$mystat m JOIN v$statname n ON n.statistic# = m.statistic# + WHERE n.name IN (${STAT_NAMES.map((_, i) => `:n${i}`).join(', ')})`, + Object.fromEntries(STAT_NAMES.map((n, i) => [`n${i}`, n])), + { outFormat: oracledb.OUT_FORMAT_OBJECT }, + ); + return Object.fromEntries(r.rows.map((x) => [x.NAME, Number(x.VALUE)])); +} diff --git a/labs/converged-modeling-patterns/app/src/gate.js b/labs/converged-modeling-patterns/app/src/gate.js new file mode 100644 index 0000000..9b3b230 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/gate.js @@ -0,0 +1,103 @@ +export class GateError extends Error { + constructor(code, status, message) { + super(message); + this.code = code; + this.status = status; + } +} + +const busy = () => new GateError('busy', 429, 'busy, retry'); +const paused = () => new GateError('paused', 423, 'paused by instructor'); + +export class Gate { + #permits; + #free; + #queueTimeoutMs; + #queue = []; + #running = new Map(); + #paused = false; + #seq = 0; + + constructor({ permits = 1, queueTimeoutMs = 30000 } = {}) { + if (!Number.isInteger(permits) || permits < 1) throw new Error('permits must be >= 1'); + this.#permits = permits; + this.#free = permits; + this.#queueTimeoutMs = queueTimeoutMs; + } + + run(who, fn) { + if (this.#paused) return Promise.reject(paused()); + if (this.#queue.some((e) => e.userId === who.userId)) { + return Promise.reject(new GateError('already_queued', 409, 'already queued, wait for your previous request')); + } + const weight = who.exclusive ? this.#permits : 1; + return new Promise((resolve, reject) => { + const entry = { id: ++this.#seq, userId: who.userId, label: who.label, weight, fn, resolve, reject, enqueuedAt: Date.now() }; + entry.timer = setTimeout(() => { + const i = this.#queue.indexOf(entry); + if (i !== -1) { + this.#queue.splice(i, 1); + reject(busy()); + this.#pump(); + } + }, this.#queueTimeoutMs); + this.#queue.push(entry); + this.#pump(); + }); + } + + #pump() { + while (this.#queue.length && this.#queue[0].weight <= this.#free) { + const e = this.#queue.shift(); + clearTimeout(e.timer); + this.#free -= e.weight; + const ctx = {}; + this.#running.set(e.id, { id: e.id, userId: e.userId, label: e.label, startedAt: Date.now(), ctx }); + Promise.resolve() + .then(() => e.fn(ctx)) + .then(e.resolve, e.reject) + .finally(() => { + this.#running.delete(e.id); + this.#free += e.weight; + this.#pump(); + }); + } + } + + position(userId) { + const i = this.#queue.findIndex((e) => e.userId === userId); + return i === -1 ? null : i; + } + + status() { + const now = Date.now(); + return { + permits: this.#permits, + free: this.#free, + paused: this.#paused, + queued: this.#queue.map((e) => ({ userId: e.userId, label: e.label, waitingMs: now - e.enqueuedAt })), + running: [...this.#running.values()].map((r) => ({ id: r.id, userId: r.userId, label: r.label, elapsedMs: now - r.startedAt })), + }; + } + + pause(on) { + this.#paused = Boolean(on); + if (this.#paused) { + for (const e of this.#queue.splice(0)) { + clearTimeout(e.timer); + e.reject(paused()); + } + } + } + + // Sets ctx.cancelled (statement loops check it before each statement) and calls the + // operation's cancel hook. Returns whether the operation had a hook to stop it. + cancel(id) { + const r = this.#running.get(id); + if (!r) return false; + r.ctx.cancelled = true; + if (typeof r.ctx.cancel !== 'function') return false; + r.ctx.cancel(); + return true; + } +} diff --git a/labs/converged-modeling-patterns/app/src/guard.js b/labs/converged-modeling-patterns/app/src/guard.js new file mode 100644 index 0000000..0cdcd16 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/guard.js @@ -0,0 +1,52 @@ +import { sqlStageText } from './cache.js'; + +// Defence in depth only: the isolation boundary is each attendee's schema privileges. +const RULES = [ + [/\bALTER SYSTEM\b/, 'ALTER SYSTEM'], + [/\bALTER DATABASE\b/, 'ALTER DATABASE'], + [/\b(CREATE|ALTER|DROP) USER\b/, 'CREATE/ALTER/DROP USER'], + [/\bGRANT\b/, 'GRANT'], + [/\bREVOKE\b/, 'REVOKE'], + [/\bDATABASE LINK\b/, 'DATABASE LINK'], + [/\bDBMS_SCHEDULER\b/, 'DBMS_SCHEDULER'], + [/\bDBMS_JOB\b/, 'DBMS_JOB'], + [/\bUTL_[A-Z_]+/, 'UTL_* packages'], + [/\bDBMS_PIPE\b/, 'DBMS_PIPE'], + [/\bCREATE (OR REPLACE )?DIRECTORY\b/, 'CREATE DIRECTORY'], +]; + +export function normalizeForGuard(text) { + return String(text) + .replace(/\/\*[\s\S]*?\*\//g, ' ') + .replace(/--[^\n]*/g, ' ') + .toUpperCase() + .replace(/\s+/g, ' ') + .trim(); +} + +export function classifySql(text) { + const n = normalizeForGuard(text); + if (!n) return { allowed: false, reason: 'Empty statement' }; + for (const [re, label] of RULES) if (re.test(n)) return { allowed: false, reason: `Blocked in this lab: ${label}` }; + return { allowed: true }; +} + +function sqlStages(value, out = []) { + if (Array.isArray(value)) value.forEach((v) => sqlStages(v, out)); + else if (value && typeof value === 'object' && !(value instanceof RegExp) && !(value instanceof Date)) { + for (const [k, v] of Object.entries(value)) { + const sql = k === '$sql' ? sqlStageText(v) : null; + if (sql !== null) out.push(sql); // string form, or the object form's `statement` + else sqlStages(v, out); + } + } + return out; +} + +export function classifyMongo(cmd) { + for (const sql of sqlStages(cmd.args ?? [])) { + const r = classifySql(sql); + if (!r.allowed) return r; + } + return { allowed: true }; +} diff --git a/labs/converged-modeling-patterns/app/src/routes/admin.js b/labs/converged-modeling-patterns/app/src/routes/admin.js new file mode 100644 index 0000000..9766ed5 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/routes/admin.js @@ -0,0 +1,122 @@ +// app/src/routes/admin.js +import express from 'express'; +import crypto from 'node:crypto'; +import { wrap } from './api.js'; +import { sign, verify, parseCookies, cookie } from '../services/auth.js'; + +const same = (a, b) => { + const x = Buffer.from(String(a)); const y = Buffer.from(String(b)); + return x.length === y.length && crypto.timingSafeEqual(x, y); +}; + +export function adminRouter({ cfg, runner, workspaces, gate, cache, patterns, mongo, sessionSecret }) { + const r = express.Router(); + if (cfg.mode !== 'event') { + r.use((req, res) => res.status(404).json({ error: 'admin is only available in event mode' })); + return r; + } + + r.post('/login', (req, res) => { + if (!same(req.body?.password ?? '', cfg.event.adminPassword)) return res.status(403).json({ error: 'wrong password' }); + res.setHeader('Set-Cookie', cookie('lab_admin', sign('admin', sessionSecret), { maxAgeSec: 12 * 3600 })); + return res.json({ ok: true }); + }); + + r.use((req, res, next) => (verify(parseCookies(req.headers.cookie).lab_admin, sessionSecret) === 'admin' + ? next() : res.status(401).json({ error: 'admin sign-in required' }))); + + // Rebuilds every pattern in one workspace. Its own gate userId, so it never collides with + // the attendee's own queued request (the gate allows one queued request per userId). + // Returns the failures, one line per pattern; [] when every build succeeded. + const buildAll = async (w) => { + const user = { id: `admin-reset-${w.schema}`, workspace: w }; + const failed = []; + for (const p of patterns) { + try { + const out = await runner.reset({ user, patternId: p.id }); + if (!out.ok) failed.push(`${p.id}: ${out.errors[0]?.error ?? 'build failed'}`); + } catch (e) { + failed.push(`${p.id}: ${e.message}`); + } + } + return failed; + }; + + // Progress of the last "reset every attendee" (runs in the background, like prewarm). + let resets = { running: false, total: 0, done: 0, failed: [] }; + + r.get('/status', wrap(async (req, res) => res.json({ + gate: gate.status(), + cache: { size: cache.size, enabled: cache.enabled }, + timeouts: runner.timeouts, + resets, + pending: await workspaces.pendingCount(), + attendees: await workspaces.list(), + }))); + + r.post('/timeouts', (req, res) => { + const ok = (v) => Number.isInteger(v) && v >= 1000 && v <= 60000; + const sqlMs = Number(req.body?.sqlMs); const mongoMs = Number(req.body?.mongoMs); + if (!ok(sqlMs) || !ok(mongoMs)) return res.status(400).json({ error: 'sqlMs and mongoMs must be integers 1000–60000' }); + runner.timeouts = { sqlMs, mongoMs }; + return res.json(runner.timeouts); + }); + + r.post('/cancel', (req, res) => res.json({ cancelled: gate.cancel(Number(req.body?.id)) })); + + r.post('/pause', (req, res) => { gate.pause(Boolean(req.body?.on)); res.json({ paused: gate.status().paused }); }); + + r.post('/cache', (req, res) => { + cache.enabled = Boolean(req.body?.enabled); + if (!cache.enabled) cache.clear(); + res.json({ enabled: cache.enabled }); + }); + + r.post('/prewarm', (req, res) => { + const count = Number(req.body?.count); + if (!Number.isInteger(count) || count < 1 || count > 200) return res.status(400).json({ error: 'count must be 1–200' }); + (async () => { + for (let i = 0; i < count; i++) { + const w = await gate.run({ userId: `prewarm-${i}`, label: 'prewarm', exclusive: true }, () => workspaces.provision()); + await buildAll(w); + } + })().catch((e) => console.error('[lab-ui] prewarm failed', e)); + return res.status(202).json({ started: count }); + }); + + r.post('/reset-attendee', wrap(async (req, res) => { + const schema = String(req.body?.schema ?? ''); + if (schema === '*') { + if (resets.running) return res.status(409).json({ error: 'a reset of every attendee is already running' }); + const all = await workspaces.list(); + resets = { running: true, total: all.length, done: 0, failed: [] }; + const state = resets; + (async () => { + for (const a of all) { + try { + const w = await workspaces.findBySchema(a.schema); + const failed = w ? await buildAll(w) : ['workspace no longer exists']; + if (failed.length) state.failed.push({ schema: a.schema, error: failed.join('; ') }); + } catch (e) { + state.failed.push({ schema: a.schema, error: e.message }); + } + state.done += 1; + } + })().catch((e) => console.error('[lab-ui] reset-all failed', e)).finally(() => { state.running = false; }); + return res.status(202).json({ started: all.length }); + } + const w = await workspaces.findBySchema(schema); + if (!w) return res.status(404).json({ error: 'no such workspace' }); + const failed = await buildAll(w); + if (failed.length) return res.status(500).json({ ok: false, error: `reset failed: ${failed.join('; ')}`, errors: failed }); + return res.json({ ok: true, reset: 1 }); + })); + + r.post('/end-event', wrap(async (req, res) => { + const { dropped, pending } = await gate.run({ userId: 'admin', label: 'end event', exclusive: true }, () => workspaces.dropAll(mongo)); + cache.clear(); + return res.json({ dropped, pending }); + })); + + return r; +} diff --git a/labs/converged-modeling-patterns/app/src/routes/api.js b/labs/converged-modeling-patterns/app/src/routes/api.js new file mode 100644 index 0000000..825958f --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/routes/api.js @@ -0,0 +1,92 @@ +// app/src/routes/api.js +import express from 'express'; +import { GateError } from '../gate.js'; +import { sign, verify, parseCookies, cookie } from '../services/auth.js'; + +const DAY = 24 * 3600; + +export function wrap(fn) { + return (req, res, next) => Promise.resolve(fn(req, res, next)).catch((err) => { + if (err instanceof GateError) return res.status(err.status).json({ error: err.message, code: err.code }); + console.error('[lab-ui]', err); + return res.status(500).json({ error: 'internal error' }); + }); +} + +export function apiRouter({ cfg, runner, workspaces, gate, patterns, sessionSecret }) { + const r = express.Router(); + const publicPaths = new Set(['/config', '/signin', '/admin/login']); + + r.use(wrap(async (req, res, next) => { + if (cfg.mode === 'solo') { + const w = workspaces.solo(); + req.user = { id: 'solo', workspace: w, profile: { schema: w.schema, email: null, name: null } }; + return next(); + } + if (publicPaths.has(req.path) || req.path.startsWith('/admin/')) return next(); + const schema = verify(parseCookies(req.headers.cookie).lab_sid, sessionSecret); + const w = schema ? await workspaces.findBySchema(schema) : null; + if (!w) return res.status(401).json({ error: 'sign in first' }); + req.user = { id: w.schema, workspace: w, profile: { schema: w.schema, email: w.email, name: w.name } }; + return next(); + })); + + r.get('/config', (req, res) => res.json({ mode: cfg.mode, eventCodeRequired: cfg.mode === 'event' && Boolean(cfg.event.code) })); + + r.post('/signin', wrap(async (req, res) => { + if (cfg.mode !== 'event') return res.status(404).json({ error: 'not in event mode' }); + const { name, email, code } = req.body ?? {}; + if (!name?.trim() || !/^[^@\s]+@[^@\s]+$/.test(email ?? '')) return res.status(400).json({ error: 'name and a valid email are required' }); + if (cfg.event.code && code !== cfg.event.code) return res.status(403).json({ error: 'wrong event code' }); + const w = await gate.run({ userId: email.toLowerCase(), label: 'sign-in', exclusive: true }, () => workspaces.assign({ email, name: name.trim() })); + res.setHeader('Set-Cookie', cookie('lab_sid', sign(w.schema, sessionSecret), { maxAgeSec: 7 * DAY })); + return res.json({ schema: w.schema }); + })); + + r.get('/me', (req, res) => res.json({ mode: cfg.mode, user: req.user.profile })); + + r.get('/patterns', (req, res) => res.json(patterns.map((p) => ({ + id: p.id, + meta: p.meta, + cards: { + document: p.lanes.document.map(({ title, notes, sql, measure, help }) => ({ title, notes, sql, measure, help })), + converged: p.lanes.converged.map(({ title, notes, sql, measure, help }) => ({ title, notes, sql, measure, help })), + mongo: p.lanes.mongo.map(({ title, notes, command, help }) => ({ title, notes, command, help })), + }, + measures: p.measures.map((m) => ({ tag: m.tag, documentSql: m.document.sql, convergedSql: m.converged.sql, + help: { document: m.document.help, converged: m.converged.help } })), + })))); + + const validPattern = (id) => id === null || id === undefined || patterns.some((p) => p.id === id); + + r.post('/run', wrap(async (req, res) => { + const { lane, patternId = null, text } = req.body ?? {}; + if (!['sql', 'mongo'].includes(lane) || typeof text !== 'string' || !validPattern(patternId)) { + return res.status(400).json({ error: 'lane must be "sql" or "mongo", text a string, patternId a known pattern or null' }); + } + const out = lane === 'sql' + ? await runner.runSql({ user: req.user, patternId, text }) + : await runner.runMongoText({ user: req.user, patternId, text }); + return res.json(out); + })); + + r.get('/queue', (req, res) => { + const s = gate.status(); + res.json({ position: gate.position(req.user.id), depth: s.queued.length, paused: s.paused }); + }); + + r.post('/measure', wrap(async (req, res) => { + const { patternId, tag } = req.body ?? {}; + const p = patterns.find((x) => x.id === patternId); + if (!p || !p.measures.some((m) => m.tag === tag)) return res.status(400).json({ error: 'unknown pattern or measure tag' }); + return res.json(await runner.measure({ user: req.user, patternId, tag })); + })); + + r.post('/reset', wrap(async (req, res) => { + const { patternId } = req.body ?? {}; + if (!patterns.some((p) => p.id === patternId)) return res.status(400).json({ error: 'unknown pattern' }); + return res.json(await runner.reset({ user: req.user, patternId })); + })); + + return r; +} diff --git a/labs/converged-modeling-patterns/app/src/routes/figures.js b/labs/converged-modeling-patterns/app/src/routes/figures.js new file mode 100644 index 0000000..4ccb673 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/routes/figures.js @@ -0,0 +1,26 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import express from 'express'; + +const NAME = /^[a-z0-9-]+\.svg$/; + +// Serves exported deck figures from patterns//figures/. The pattern id must be a loaded +// pattern and the file a plain lower-case .svg name, so nothing outside that folder is reachable. +export function figuresRouter({ cfg, patterns }) { + const r = express.Router(); + const ids = new Set(patterns.map((p) => p.id)); + r.get('/:pattern/:file', (req, res) => { + const { pattern, file } = req.params; + if (!ids.has(pattern) || !NAME.test(file)) return res.status(404).end(); + const full = path.join(cfg.patternsDir, pattern, 'figures', file); + if (!fs.existsSync(full)) return res.status(404).end(); + res.set('Content-Type', 'image/svg+xml'); + res.set('X-Content-Type-Options', 'nosniff'); + res.set('Cache-Control', 'public, max-age=3600'); + // Opened full size in its own tab an SVG is a document, so lock it down: no scripts, no loads. + res.set('Content-Security-Policy', "default-src 'none'; style-src 'unsafe-inline'"); + // A file removed between existsSync and the read (or any read error) is a 404, not a 500. + return res.sendFile(full, (err) => { if (err && !res.headersSent) res.status(404).end(); }); + }); + return r; +} diff --git a/labs/converged-modeling-patterns/app/src/server.js b/labs/converged-modeling-patterns/app/src/server.js new file mode 100644 index 0000000..1e13725 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/server.js @@ -0,0 +1,74 @@ +// app/src/server.js +import path from 'node:path'; +import express from 'express'; +import { loadConfig } from './config.js'; +import { bootstrap, createPools } from './db/oracle.js'; +import { MongoPool } from './db/mongo.js'; +import { Gate } from './gate.js'; +import { ResultCache } from './cache.js'; +import { loadPatterns } from './content/patterns.js'; +import { Workspaces } from './services/workspaces.js'; +import { Runner } from './services/runner.js'; +import { apiRouter } from './routes/api.js'; +import { adminRouter } from './routes/admin.js'; +import { figuresRouter } from './routes/figures.js'; + +export function createApp(deps) { + const app = express(); + app.disable('x-powered-by'); + app.use(express.json({ limit: '64kb' })); + app.use('/api/admin', adminRouter(deps)); + app.use('/api', apiRouter(deps)); + app.use('/figures', figuresRouter(deps)); + app.use(express.static(path.resolve(import.meta.dirname, '../public'), { maxAge: '1h' })); + app.use((err, req, res, next) => { // body-parser errors etc. + if (res.headersSent) return next(err); + return res.status(err.status ?? 400).json({ error: err.type === 'entity.too.large' ? 'request too large' : 'bad request' }); + }); + return app; +} + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + +async function waitForBootstrap(cfg) { + for (let i = 1; i <= 60; i++) { + try { + await bootstrap(cfg); + if (cfg.mode !== 'event') return; + const pools = await createPools(cfg); + const c = await pools.control.getConnection(); + const r = await c.execute("SELECT COUNT(*) FROM user_role_privs WHERE granted_role = 'ORDS_ADMINISTRATOR_ROLE'"); + await c.close(); await pools.close(); + if (r.rows[0][0] > 0) return; + console.log('[lab-ui] waiting for ORDS to be installed…'); + } catch (e) { + console.log(`[lab-ui] database not ready (${i}/60): ${e.message}`); + } + await sleep(10000); + } + throw new Error('database did not become ready in 10 minutes'); +} + +export async function main() { + const cfg = loadConfig(); + await waitForBootstrap(cfg); + const pools = await createPools(cfg); + const mongo = new MongoPool(cfg); + const gate = new Gate(cfg.gate); + const cache = new ResultCache(cfg.cache); + cache.enabled = cfg.cache.enabled; + const patterns = loadPatterns(cfg.patternsDir); + const workspaces = new Workspaces({ cfg, pools }); + await workspaces.load(); + if (cfg.mode === 'event') { + workspaces.startReaper(30000, (fn) => gate.run({ userId: 'reaper', label: 'reap', exclusive: true }, fn)); + } + const runner = new Runner({ cfg, gate, cache, pools, mongo, workspaces, patterns }); + const sessionSecret = await workspaces.sessionSecret(); + const app = createApp({ cfg, runner, workspaces, gate, cache, patterns, mongo, sessionSecret }); + app.listen(cfg.port, () => console.log(`[lab-ui] ${cfg.mode} mode on :${cfg.port} · ${patterns.length} patterns`)); +} + +if (process.argv[1] === import.meta.filename) { + main().catch((e) => { console.error(e); process.exit(1); }); +} diff --git a/labs/converged-modeling-patterns/app/src/services/auth.js b/labs/converged-modeling-patterns/app/src/services/auth.js new file mode 100644 index 0000000..950f3a3 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/services/auth.js @@ -0,0 +1,31 @@ +// app/src/services/auth.js +import crypto from 'node:crypto'; + +const mac = (value, secret) => crypto.createHmac('sha256', secret).update(value).digest('base64url'); + +export function sign(value, secret) { + return `${value}.${mac(value, secret)}`; +} + +export function verify(token, secret) { + if (typeof token !== 'string') return null; + const i = token.lastIndexOf('.'); + if (i <= 0) return null; + const value = token.slice(0, i); + const a = Buffer.from(token.slice(i + 1)); + const b = Buffer.from(mac(value, secret)); + return a.length === b.length && crypto.timingSafeEqual(a, b) ? value : null; +} + +export function parseCookies(header) { + const out = {}; + for (const part of String(header ?? '').split(';')) { + const i = part.indexOf('='); + if (i > 0) out[part.slice(0, i).trim()] = decodeURIComponent(part.slice(i + 1).trim()); + } + return out; +} + +export function cookie(name, value, { maxAgeSec }) { + return `${name}=${encodeURIComponent(value)}; Max-Age=${maxAgeSec}; Path=/; HttpOnly; SameSite=Lax`; +} diff --git a/labs/converged-modeling-patterns/app/src/services/runner.js b/labs/converged-modeling-patterns/app/src/services/runner.js new file mode 100644 index 0000000..90aef77 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/services/runner.js @@ -0,0 +1,241 @@ +// app/src/services/runner.js +import { performance } from 'node:perf_hooks'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { splitConsoleSql } from '../content/sqlParser.js'; +import { parseMongoCommand, MongoParseError } from '../content/mongoCommand.js'; +import { classifySql, classifyMongo } from '../guard.js'; +import { isReadOnlySql, isReadOnlyMongo, patternsTouched } from '../cache.js'; +import { executeSql, readStats } from '../db/oracle.js'; +import { runMongo } from '../db/mongo.js'; + +const isTimeout = (r) => r.kind === 'error' && /DPI-1067|NJS-123|call timeout|timed out/i.test(r.error); +const IDLE_POLL_MS = 250; +const IDLE_CEILING_MS = 60000; +const labErr = (code, error) => ({ kind: 'error', code, error, elapsedMs: 0 }); +// Reset/build (system operations) get one 120 s budget for all their statements. +const SYSTEM_BUDGET_MS = 120000; + +export class Runner { + #cfg; #gate; #cache; #pools; #mongo; #ws; #patterns; #ids; #cards; + + constructor({ cfg, gate, cache, pools, mongo, workspaces, patterns }) { + this.#cfg = cfg; + this.#gate = gate; + this.#cache = cache; + this.#pools = pools; + this.#mongo = mongo; + this.#ws = workspaces; + this.#patterns = new Map(patterns.map((p) => [p.id, p])); + this.#ids = patterns.map((p) => p.id); + // Cache eligibility: only a pattern's own card text (trimmed) is ever cached. + this.#cards = new Map(patterns.map((p) => [p.id, { + sql: new Set([...p.lanes.document, ...p.lanes.converged].map((s) => s.sql.trim())), + mongo: new Set(p.lanes.mongo.map((c) => c.command.trim())), + }])); + this.timeouts = { sqlMs: cfg.gate.sqlTimeoutMs, mongoMs: cfg.gate.mongoTimeoutMs }; + } + + // Runs `stmts` in order on one connection under ONE deadline of `budgetMs`: each + // statement's callTimeout is the time remaining. Before each statement, a cancelled run + // (ctx.cancelled, set by Gate.cancel) yields LAB-CANCELLED for it and every later one; an + // exhausted budget, or a statement stopped by its callTimeout, yields LAB-TIMEOUT likewise. + async #runStatements(conn, ctx, track, stmts, budgetMs) { + const deadline = performance.now() + budgetMs; + const secs = budgetMs / 1000; + const out = []; + let stop = null; + for (const sql of stmts) { + if (!stop && ctx.cancelled) stop = labErr('LAB-CANCELLED', 'Run was cancelled; this statement did not run'); + const left = Math.floor(deadline - performance.now()); + if (!stop && left <= 0) stop = labErr('LAB-TIMEOUT', `Run exceeded ${secs} s; this statement did not run`); + if (stop) { out.push(stop); continue; } + const r = track(await executeSql(conn, sql, { ...this.#limits(), timeoutMs: left })); + if (isTimeout(r)) { + out.push({ kind: 'error', code: 'LAB-TIMEOUT', error: `Run exceeded ${secs} s and was stopped`, elapsedMs: r.elapsedMs }); + stop = labErr('LAB-TIMEOUT', `Run exceeded ${secs} s; this statement did not run`); + continue; + } + out.push(r); + } + return out; + } + + #limits(autoCommit = true) { + const { maxRows, maxBytes } = this.#cfg.limits; + return { timeoutMs: this.timeouts.sqlMs, maxRows, maxBytes, autoCommit }; + } + + // A run is cacheable only when its text is exactly one of the pattern's cards (after + // trimming leading/trailing whitespace), it is read-only, and the workspace is pristine. + #cacheKey(p, lane, text, readOnly, ws) { + if (!p || !readOnly || !this.#cards.get(p.id)[lane].has(text.trim())) return null; + if (this.#ws.isDirty(ws.schema, p.id)) return null; + return this.#cache.key(p.id, text, p.version); + } + + #pattern(patternId) { + const p = patternId ? this.#patterns.get(patternId) : null; + if (patternId && !p) throw new Error(`unknown pattern ${patternId}`); + return p; + } + + async #withConn(ws, ctx, fn) { + const conn = await this.#pools.exec.getConnection({ user: ws.schema, password: ws.password }); + ctx.cancel = () => { conn.break().catch(() => {}); }; + let drop = false; + try { + const out = await fn(conn, (r) => { if (isTimeout(r)) drop = true; return r; }); + return out; + } finally { + await conn.close({ drop }).catch(() => {}); + } + } + + async ensureBuilt(user, patternId) { + const p = this.#pattern(patternId); + if (!p) return; + if (this.#ws.builtVersion(user.workspace.schema, patternId) === p.version) return; + const r = await this.reset({ user, patternId }); + if (!r.ok) throw new Error(`could not build ${patternId}: ${r.errors[0]?.error}`); + } + + async reset({ user, patternId }) { + const p = this.#pattern(patternId); + const ws = user.workspace; + const errors = await this.#gate.run({ userId: user.id, label: `${patternId} · reset`, exclusive: true }, (ctx) => + this.#withConn(ws, ctx, async (conn, track) => { + const stmts = [...p.setup.document, ...p.setup.converged].map((st) => st.sql); + const results = await this.#runStatements(conn, ctx, track, stmts, SYSTEM_BUDGET_MS); + return results.flatMap((r, i) => (r.kind === 'error' ? [{ sql: stmts[i], error: r.error, code: r.code }] : [])); + })); + if (!errors.length) { + await this.#ws.markBuilt(ws.schema, patternId, p.version); + await this.#ws.clearDirty(ws.schema, patternId); + } + return { ok: errors.length === 0, errors }; + } + + async runSql({ user, patternId, text }) { + const stmts = splitConsoleSql(text); + if (!stmts.length) return { lane: 'sql', results: [labErr('LAB-PARSE', 'Nothing to run')], cached: false }; + if (stmts.length > this.#cfg.limits.maxStatements) { + return { lane: 'sql', results: [labErr('LAB-PARSE', `At most ${this.#cfg.limits.maxStatements} statements per run`)], cached: false }; + } + for (const s of stmts) { + const g = classifySql(s); + if (!g.allowed) return { lane: 'sql', results: [labErr('LAB-GUARD', g.reason)], cached: false }; + } + const p = this.#pattern(patternId); + if (p) await this.ensureBuilt(user, patternId); + const ws = user.workspace; + const readOnly = isReadOnlySql(stmts); + const key = this.#cacheKey(p, 'sql', text, readOnly, ws); + const hit = key ? this.#cache.get(key) : undefined; + if (hit) return { lane: 'sql', results: hit, cached: true }; + + const results = await this.#gate.run({ userId: user.id, label: `${patternId ?? 'console'} · SQL` }, (ctx) => + this.#withConn(ws, ctx, (conn, track) => this.#runStatements(conn, ctx, track, stmts, this.timeouts.sqlMs))); + if (!readOnly) await this.#ws.markDirty(ws.schema, patternsTouched(text, this.#ids)); + if (key && results.every((r) => r.kind !== 'error')) this.#cache.set(key, results); + return { lane: 'sql', results, cached: false }; + } + + async runMongoText({ user, patternId, text }) { + let cmd; + try { + cmd = parseMongoCommand(text); + } catch (e) { + if (e instanceof MongoParseError) return { lane: 'mongo', results: [labErr('LAB-PARSE', e.message)], cached: false }; + throw e; + } + const g = classifyMongo(cmd); + if (!g.allowed) return { lane: 'mongo', results: [labErr('LAB-GUARD', g.reason)], cached: false }; + const p = this.#pattern(patternId); + if (p) await this.ensureBuilt(user, patternId); + const ws = user.workspace; + const readOnly = isReadOnlyMongo(cmd); + const key = this.#cacheKey(p, 'mongo', text, readOnly, ws); + const hit = key ? this.#cache.get(key) : undefined; + if (hit) return { lane: 'mongo', results: hit, cached: true }; + + const { maxRows, maxBytes } = this.#cfg.limits; + const result = await this.#gate.run({ userId: user.id, label: `${patternId ?? 'console'} · MongoDB` }, async (ctx) => { + // The Oracle API for MongoDB ignores maxTimeMS, so the runner enforces the timeout itself. + let stop; + const stopped = new Promise((resolve) => { stop = resolve; }); + ctx.cancel = () => stop('cancel'); + const timeoutMs = this.timeouts.mongoMs; + const timer = setTimeout(() => stop('timeout'), timeoutMs); + const start = performance.now(); + try { + const client = this.#mongo.client(ws.schema, ws.password); + const op = runMongo(client, ws.schema.toUpperCase(), cmd, { timeoutMs, maxRows, maxBytes }); + const won = await Promise.race([op.then((r) => ({ r })), stopped.then((why) => ({ why }))]); + if (won.r) return won.r; + await this.#mongo.drop(ws.schema); + await this.#waitIdle(ws.schema); + const elapsedMs = Math.round(performance.now() - start); + return won.why === 'timeout' + ? { kind: 'error', code: 'LAB-TIMEOUT', error: `MongoDB operation exceeded ${timeoutMs / 1000} s and was stopped`, elapsedMs } + : { kind: 'error', code: 'LAB-CANCELLED', error: 'MongoDB operation was cancelled', elapsedMs }; + } finally { + clearTimeout(timer); + } + }); + if (!readOnly) await this.#ws.markDirty(ws.schema, patternsTouched(`${cmd.collection ?? ''} ${text}`, this.#ids)); + if (key && result.kind !== 'error') this.#cache.set(key, [result]); + return { lane: 'mongo', results: [result], cached: false }; + } + + // Hold the gate permit until the database shows no ACTIVE session for the schema (60 s ceiling). + async #waitIdle(schema) { + const s = schema.toUpperCase(); + const deadline = Date.now() + IDLE_CEILING_MS; + while (Date.now() < deadline) { + const c = await this.#pools.control.getConnection(); + let active; + try { + const r = await c.execute("SELECT COUNT(*) FROM v$session WHERE username = :s AND status = 'ACTIVE'", { s }); + active = Number(r.rows[0][0]); + } finally { + await c.close(); + } + if (active === 0) return; + await sleep(IDLE_POLL_MS); + } + console.warn(`[lab-ui] mongo op for ${s} still active after 60 s — releasing the queue`); + } + + async measure({ user, patternId, tag }) { + const p = this.#pattern(patternId); + const pair = p?.measures.find((m) => m.tag === tag); + if (!pair) throw new Error(`unknown measure ${patternId}/${tag}`); + await this.ensureBuilt(user, patternId); + const ws = user.workspace; + return this.#gate.run({ userId: user.id, label: `${patternId} · measure ${tag}`, exclusive: true }, (ctx) => + this.#withConn(ws, ctx, async (conn, track) => { + // In-memory undo / private redo strands defer 'redo size' and 'db block changes' + // in V$MYSTAT until commit or rollback, so a read before the rollback sees 0 (or a + // stale value). Turning it off makes each side's deltas exact and repeatable. + // No reset needed: exec-pool connections are standalone and closed after this call. + try { + await conn.execute('ALTER SESSION SET "_in_memory_undo" = false'); + } catch (e) { + const err = labErr('LAB-MEASURE', `could not prepare the measurement session: ${e.message}`); + return { tag, document: { sql: pair.document.sql, stats: {}, result: err }, converged: { sql: pair.converged.sql, stats: {}, result: err } }; + } + const side = async (stmt) => { + // Unmeasured warm-up pass: parse, cursor and first-touch effects stay out of the numbers. + track(await executeSql(conn, stmt.sql, this.#limits(false))); + await conn.rollback(); + const before = await readStats(conn); + const result = track(await executeSql(conn, stmt.sql, this.#limits(false))); + const after = await readStats(conn); + await conn.rollback(); + const stats = Object.fromEntries(Object.keys(after).map((n) => [n, after[n] - before[n]])); + return { sql: stmt.sql, stats, result }; + }; + return { tag, document: await side(pair.document), converged: await side(pair.converged) }; + })); + } +} diff --git a/labs/converged-modeling-patterns/app/src/services/workspaces.js b/labs/converged-modeling-patterns/app/src/services/workspaces.js new file mode 100644 index 0000000..5f59745 --- /dev/null +++ b/labs/converged-modeling-patterns/app/src/services/workspaces.js @@ -0,0 +1,295 @@ +// app/src/services/workspaces.js +import oracledb from 'oracledb'; +import crypto from 'node:crypto'; + +const OBJ = { outFormat: oracledb.OUT_FORMAT_OBJECT }; +const k = (schema, patternId) => `${schema}|${patternId}`; +const WS_NAME = /^WS_[0-9A-F]{6}$/; // provision()'s shape; checked before any dynamic DDL on a parked name + +export class Workspaces { + cfg; + pools; + #dirty = new Set(); + #built = new Map(); + + constructor({ cfg, pools }) { + this.cfg = cfg; + this.pools = pools; + } + + async withControl(fn) { + const c = await this.pools.control.getConnection(); + try { + return await fn(c); + } finally { + await c.close(); + } + } + + async load() { + await this.withControl(async (c) => { + this.#dirty.clear(); + this.#built.clear(); + (await c.execute('SELECT schema_name, pattern_id FROM lab_dirty', [], OBJ)).rows + .forEach((r) => this.#dirty.add(k(r.SCHEMA_NAME, r.PATTERN_ID))); + (await c.execute('SELECT schema_name, pattern_id, version FROM lab_built', [], OBJ)).rows + .forEach((r) => this.#built.set(k(r.SCHEMA_NAME, r.PATTERN_ID), r.VERSION)); + }); + } + + solo() { + return { userId: 'solo', schema: this.cfg.db.cmpUser, password: this.cfg.db.cmpPassword }; + } + + isDirty(schema, patternId) { + return this.#dirty.has(k(schema, patternId)); + } + + async markDirty(schema, patternIds) { + const add = patternIds.filter((p) => !this.isDirty(schema, p)); + if (!add.length) return; + await this.withControl((c) => c.executeMany( + `MERGE INTO lab_dirty d USING (SELECT :s AS s, :p AS p FROM dual) x + ON (d.schema_name = x.s AND d.pattern_id = x.p) + WHEN NOT MATCHED THEN INSERT (schema_name, pattern_id) VALUES (x.s, x.p)`, + add.map((p) => ({ s: schema, p })), { autoCommit: true })); + add.forEach((p) => this.#dirty.add(k(schema, p))); + } + + async clearDirty(schema, patternId) { + await this.withControl((c) => c.execute('DELETE FROM lab_dirty WHERE schema_name = :s AND pattern_id = :p', + { s: schema, p: patternId }, { autoCommit: true })); + this.#dirty.delete(k(schema, patternId)); + } + + builtVersion(schema, patternId) { + return this.#built.get(k(schema, patternId)) ?? null; + } + + async markBuilt(schema, patternId, version) { + await this.withControl((c) => c.execute( + `MERGE INTO lab_built b USING (SELECT :s AS s, :p AS p, :v AS v FROM dual) x + ON (b.schema_name = x.s AND b.pattern_id = x.p) + WHEN MATCHED THEN UPDATE SET b.version = x.v + WHEN NOT MATCHED THEN INSERT (schema_name, pattern_id, version) VALUES (x.s, x.p, x.v)`, + { s: schema, p: patternId, v: version }, { autoCommit: true })); + this.#built.set(k(schema, patternId), version); + } + + forgetInMemory(schema) { + for (const key of [...this.#dirty]) if (key.startsWith(`${schema}|`)) this.#dirty.delete(key); + for (const key of [...this.#built.keys()]) if (key.startsWith(`${schema}|`)) this.#built.delete(key); + } + + async sessionSecret() { + return this.withControl(async (c) => { + const r = await c.execute("SELECT v FROM lab_settings WHERE k = 'session_secret'", [], OBJ); + if (r.rows.length) return r.rows[0].V; + const v = crypto.randomBytes(32).toString('hex'); + await c.execute( + `MERGE INTO lab_settings s USING (SELECT 'session_secret' AS k, :v AS v FROM dual) x ON (s.k = x.k) + WHEN NOT MATCHED THEN INSERT (k, v) VALUES (x.k, x.v)`, { v }, { autoCommit: true }); + return (await c.execute("SELECT v FROM lab_settings WHERE k = 'session_secret'", [], OBJ)).rows[0].V; + }); + } + + #row(r) { + return r ? { userId: r.SCHEMA_NAME, schema: r.SCHEMA_NAME, password: r.SCHEMA_PASSWORD, email: r.EMAIL, name: r.DISPLAY_NAME } : null; + } + + async findByEmail(email) { + return this.withControl(async (c) => this.#row((await c.execute( + 'SELECT * FROM lab_users WHERE email = :e', { e: email.trim().toLowerCase() }, OBJ)).rows[0])); + } + + async findBySchema(schema) { + return this.withControl(async (c) => this.#row((await c.execute( + 'SELECT * FROM lab_users WHERE schema_name = :s', { s: schema }, OBJ)).rows[0])); + } + + async provision({ email = null, name = null } = {}) { + const schema = `WS_${crypto.randomBytes(3).toString('hex').toUpperCase()}`; + const password = `Ws${crypto.randomBytes(12).toString('hex')}`; + await this.withControl(async (c) => { + const ddl = [ + `CREATE USER ${schema} IDENTIFIED BY "${password}" QUOTA 50M ON users`, + `GRANT CREATE SESSION, CREATE TABLE, CREATE VIEW, CREATE PROCEDURE, CREATE TRIGGER, CREATE SEQUENCE, CREATE MATERIALIZED VIEW, CREATE PROPERTY GRAPH, SODA_APP TO ${schema}`, + `GRANT SELECT ON sys.v_$mystat TO ${schema}`, + `GRANT SELECT ON sys.v_$statname TO ${schema}`, + `BEGIN ORDS_METADATA.ORDS_ADMIN.ENABLE_SCHEMA(p_enabled => TRUE, p_schema => '${schema}', p_url_mapping_type => 'BASE_PATH', p_url_mapping_pattern => '${schema.toLowerCase()}', p_auto_rest_auth => FALSE); COMMIT; END;`, + ]; + for (const s of ddl) await c.execute(s); + await c.execute( + `INSERT INTO lab_users (schema_name, schema_password, email, display_name, assigned_at) + VALUES (:s, :p, :e, :n, CASE WHEN :e IS NULL THEN NULL ELSE SYSTIMESTAMP END)`, + { s: schema, p: password, e: email ? email.trim().toLowerCase() : null, n: name }, { autoCommit: true }); + }); + return { userId: schema, schema, password, email, name }; + } + + async assign({ email, name }) { + const existing = await this.findByEmail(email); + if (existing) return existing; + const claimed = await this.withControl(async (c) => { + // FOR UPDATE cannot be combined with FETCH FIRST (it is implemented via an analytic + // ROW_NUMBER() internally, which Oracle rejects with ORA-02014, same as DISTINCT/GROUP BY). + // An ORDER BY inside the locked inline view hits the same restriction, so this picks an + // arbitrary unassigned row (ROWNUM = 1, no ORDER BY) rather than strictly the oldest one — + // acceptable here since prewarmed workspaces are interchangeable. + const r = await c.execute( + `SELECT schema_name FROM ( + SELECT schema_name FROM lab_users WHERE email IS NULL + ) WHERE ROWNUM = 1 FOR UPDATE SKIP LOCKED`, [], OBJ); + if (!r.rows.length) return null; + await c.execute('UPDATE lab_users SET email = :e, display_name = :n, assigned_at = SYSTIMESTAMP WHERE schema_name = :s', + { e: email.trim().toLowerCase(), n: name, s: r.rows[0].SCHEMA_NAME }, { autoCommit: true }); + return r.rows[0].SCHEMA_NAME; + }); + return claimed ? this.findBySchema(claimed) : this.provision({ email, name }); + } + + async list() { + return this.withControl(async (c) => (await c.execute( + `SELECT u.schema_name, u.email, u.display_name, u.created_at, u.assigned_at, + (SELECT LISTAGG(b.pattern_id, ',') WITHIN GROUP (ORDER BY b.pattern_id) FROM lab_built b WHERE b.schema_name = u.schema_name) AS built + FROM lab_users u ORDER BY u.created_at`, [], OBJ)).rows.map((r) => ({ + schema: r.SCHEMA_NAME, email: r.EMAIL, name: r.DISPLAY_NAME, createdAt: r.CREATED_AT, assignedAt: r.ASSIGNED_AT, + built: r.BUILT ? r.BUILT.split(',') : [], + }))); + } + + // End event: every workspace is cut off immediately (ACCOUNT LOCK — no new connections + // possible, whatever the DB session behind it is doing) and moved out of lab_users (so + // list()/findByEmail/findBySchema stop seeing it) into lab_pending_drop, then this returns. + // The actual DROP USER happens in reapPending(), run by the background reaper + // (startReaper, first pass within its 30 s interval) — some Oracle API for MongoDB + // sessions stay open well past the client disconnecting (see task-11-report.md), so + // DROP USER ... CASCADE can't be relied on to succeed while the gate is held. + async dropAll(mongoPool) { + return this.park((await this.list()).map((u) => u.schema), mongoPool); + } + + // Lock + park the named workspaces (dropAll's body; the integration tests use it to clean + // up only the workspaces they created). A name that isn't WS_-shaped is skipped and logged + // before any dynamic DDL. Returns { dropped: 0, pending: }. + async park(schemas, mongoPool) { + const ok = schemas.filter((s) => { + if (WS_NAME.test(s)) return true; + console.warn(`[lab-ui] dropAll: skipped "${s}": not a WS_ workspace name`); + return false; + }); + await Promise.all(ok.map((s) => mongoPool?.drop(s))); + await this.withControl(async (c) => { + for (const s of ok) { + await c.execute(`ALTER USER ${s} ACCOUNT LOCK`).catch((e) => { if (e.errorNum !== 1918) throw e; }); + await c.execute( + `MERGE INTO lab_pending_drop d USING (SELECT :s AS s FROM dual) x ON (d.schema_name = x.s) + WHEN NOT MATCHED THEN INSERT (schema_name) VALUES (x.s)`, { s }); + await c.execute('DELETE FROM lab_dirty WHERE schema_name = :s', { s }); + await c.execute('DELETE FROM lab_built WHERE schema_name = :s', { s }); + await c.execute('DELETE FROM lab_users WHERE schema_name = :s', { s }); + await c.commit(); + this.forgetInMemory(s); + } + }); + return { dropped: 0, pending: ok.length }; + } + + async pendingCount() { + return this.withControl(async (c) => Number((await c.execute('SELECT COUNT(*) AS n FROM lab_pending_drop', [], OBJ)).rows[0].N)); + } + + // One pass over lab_pending_drop: DROP USER ... CASCADE each row. ORA-01918 (user + // already gone) counts as dropped. ORA-01940 (still connected) leaves the row in place + // with last_error recorded, for a later pass to retry — unless the row has been parked for + // KILL_AFTER_SEC or longer: then the schema's sessions are killed and the DROP is retried + // once (ORDS pools hold idle sessions indefinitely, so waiting alone can wait forever). + // Schema names are validated before any dynamic DDL; a row that isn't WS_-shaped, or any + // other per-row error, is recorded in last_error and the loop moves on. Idempotent and + // safe to call repeatedly/concurrently: the control pool has a single connection, so + // calls serialize, and re-attempting a DROP that already succeeded just yields ORA-01918. + static KILL_AFTER_SEC = 120; + + // `only` (optional) restricts the pass to those schema names — the integration tests + // reap just the workspaces they parked, never a live event's. + async reapPending(only = null) { + return this.withControl(async (c) => { + const filter = only ? ` WHERE schema_name IN (${only.map((_, i) => `:s${i}`).join(', ') || 'NULL'})` : ''; + const binds = only ? Object.fromEntries(only.map((s, i) => [`s${i}`, s])) : []; + const rows = (await c.execute( + `SELECT schema_name, (CAST(SYSTIMESTAMP AS DATE) - CAST(requested_at AS DATE)) * 86400 AS parked_sec + FROM lab_pending_drop${filter}`, binds, OBJ)).rows; + const note = (schema, m) => c.execute('UPDATE lab_pending_drop SET last_error = :m WHERE schema_name = :s', + { m: String(m).slice(0, 400), s: schema }, { autoCommit: true }); + const drop = async (schema) => { + try { await c.execute(`DROP USER ${schema} CASCADE`); } catch (e) { if (e.errorNum !== 1918) throw e; } + }; + let dropped = 0; + for (const r of rows) { + const schema = r.SCHEMA_NAME; + try { + if (!WS_NAME.test(schema)) { await note(schema, 'skipped: not a WS_ workspace name'); continue; } + try { + await drop(schema); + } catch (e) { + if (e.errorNum !== 1940) throw e; + if (!(r.PARKED_SEC >= Workspaces.KILL_AFTER_SEC)) { await note(schema, e.message); continue; } + const killed = await this.#killSessions(c, schema); + console.log(`[lab-ui] reapPending: killed ${killed} session(s) holding ${schema} after ${Math.round(r.PARKED_SEC)} s parked`); + await note(schema, `killed ${killed} session(s) after ${Math.round(r.PARKED_SEC)} s parked; retrying drop`); + await drop(schema); // the one retry; a failure lands in last_error below + } + await c.execute('DELETE FROM lab_pending_drop WHERE schema_name = :s', { s: schema }, { autoCommit: true }); + dropped++; + } catch (e) { + await note(schema, e.message).catch(() => {}); + } + } + return { dropped, pending: rows.length - dropped }; + }); + } + + // Kills every session of `schema` (already validated) and waits, bounded, for them to go. + async #killSessions(c, schema) { + const sessions = (await c.execute( + "SELECT sid, serial# AS serial FROM v$session WHERE username = :s AND status <> 'KILLED'", { s: schema }, OBJ)).rows; + for (const { SID, SERIAL } of sessions) { + await c.execute(`ALTER SYSTEM KILL SESSION '${Number(SID)},${Number(SERIAL)}' IMMEDIATE`) + // ORA-00030: already gone; ORA-00031: already marked for kill + .catch((e) => { if (e.errorNum !== 30 && e.errorNum !== 31) throw e; }); + } + for (let i = 0; i < 20; i++) { + const n = (await c.execute('SELECT COUNT(*) AS n FROM v$session WHERE username = :s', { s: schema }, OBJ)).rows[0].N; + if (!n) break; + await new Promise((res) => setTimeout(res, 250)); + } + return sessions.length; + } + + // Background sweep that drops what dropAll() parked. Unref'd so + // it never keeps the process alive on its own; never throws (a DB hiccup just gets + // logged and retried on the next tick). Logs one line per pass, and only when there was + // something to report, so a quiet lab doesn't spam the log every 30s. `runPass` wraps each + // pass (the server routes it through the gate exclusively); a tick first counts + // lab_pending_drop on the control pool and takes the gate only when there is something to + // reap. A tick is skipped while the previous pass is still in flight, and a gate refusal + // (busy/paused) is just retried next tick. + startReaper(intervalMs = 30000, runPass = (fn) => fn()) { + let inFlight = false; + const timer = setInterval(async () => { + if (inFlight) return; + inFlight = true; + try { + if (!(await this.pendingCount())) return; + const { dropped, pending } = await runPass(() => this.reapPending()); + if (dropped || pending) console.log(`[lab-ui] reapPending: dropped ${dropped}, pending ${pending}`); + } catch (e) { + console.error(`[lab-ui] reapPending failed: ${e.message}`); + } finally { + inFlight = false; + } + }, intervalMs); + timer.unref?.(); + return () => clearInterval(timer); + } +} diff --git a/labs/converged-modeling-patterns/app/test/integration/env.js b/labs/converged-modeling-patterns/app/test/integration/env.js new file mode 100644 index 0000000..19ff350 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/integration/env.js @@ -0,0 +1,46 @@ +import { loadConfig } from '../../src/config.js'; + +export function testConfig(overrides = {}) { + return loadConfig({ + DB_HOST: process.env.DB_HOST ?? 'localhost', + DB_PORT: process.env.DB_PORT ?? '1522', + MONGO_HOST: process.env.MONGO_HOST ?? process.env.DB_HOST ?? 'localhost', + MONGO_PORT: process.env.MONGO_PORT ?? '27018', + ORACLE_PASSWORD: process.env.ORACLE_PASSWORD ?? 'Sandbox2026', + CMP_PASSWORD: process.env.CMP_PASSWORD ?? 'CmpUser2026', + // Pass the live LAB_ADMIN password through: bootstrap() runs ALTER USER lab_admin with it, + // so a mismatch would lock a running console out of its control pool. + LAB_ADMIN_PASSWORD: process.env.LAB_ADMIN_PASSWORD ?? 'LabAdmin2026', + ...overrides, + }, { allowDefaultSecrets: true }); +} + +// Integration tests share the lab's database with whatever else is using it (a solo lab, or +// a console someone left running). They clean up ONLY the workspaces they created: lock + +// park those, then reap just those rows. Never Workspaces.dropAll() for cleanup. +export async function dropOwn(ws, mongo, schemas) { + const own = [...new Set(schemas)].filter(Boolean); + if (!own.length) return { dropped: 0, pending: 0 }; + await ws.park(own, mongo); + return ws.reapPending(own); +} + +// Schemas in lab_users or lab_pending_drop that this test did not create. A test that must +// exercise dropAll() itself (or reset-all) skips unless this is empty. +export async function foreignRows(ws, own) { + const mine = new Set(own); + const rows = await ws.withControl(async (c) => (await c.execute( + 'SELECT schema_name FROM lab_users UNION SELECT schema_name FROM lab_pending_drop')).rows.map((r) => r[0])); + return rows.filter((s) => !mine.has(s)); +} + +// Skip the running test, with the reason, when the database holds rows it did not create. +export async function skipUnlessOnlyOwn(ctx, ws, own) { + const foreign = await foreignRows(ws, own); + if (!foreign.length) return false; + const why = `skipped: lab_users/lab_pending_drop hold ${foreign.length} workspace(s) this test did not create ` + + `(${foreign.slice(0, 5).join(', ')}${foreign.length > 5 ? ', …' : ''}); it would touch them`; + console.warn(`[test] ${ctx.task.name}: ${why}`); + ctx.skip(why); + return true; +} diff --git a/labs/converged-modeling-patterns/app/test/integration/http.test.js b/labs/converged-modeling-patterns/app/test/integration/http.test.js new file mode 100644 index 0000000..d9f464a --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/integration/http.test.js @@ -0,0 +1,145 @@ +// app/test/integration/http.test.js +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import path from 'node:path'; +import request from 'supertest'; +import { testConfig, dropOwn, skipUnlessOnlyOwn } from './env.js'; +import { bootstrap, createPools } from '../../src/db/oracle.js'; +import { MongoPool } from '../../src/db/mongo.js'; +import { Gate } from '../../src/gate.js'; +import { ResultCache } from '../../src/cache.js'; +import { loadPatterns } from '../../src/content/patterns.js'; +import { Workspaces } from '../../src/services/workspaces.js'; +import { Runner } from '../../src/services/runner.js'; +import { createApp } from '../../src/server.js'; + +const patterns = loadPatterns(path.resolve(import.meta.dirname, '../../../patterns')); +let pools; let mongo; + +async function build(env) { + const cfg = testConfig(env); + const gate = new Gate(cfg.gate); + const cache = new ResultCache(cfg.cache); + const workspaces = new Workspaces({ cfg, pools }); + await workspaces.load(); + const runner = new Runner({ cfg, gate, cache, pools, mongo, workspaces, patterns }); + const sessionSecret = await workspaces.sessionSecret(); + return { app: createApp({ cfg, runner, workspaces, gate, cache, patterns, mongo, sessionSecret }), gate, workspaces }; +} + +beforeAll(async () => { + const cfg = testConfig(); + await bootstrap(cfg); + pools = await createPools(cfg); + mongo = new MongoPool(cfg); +}, 300000); +afterAll(async () => { await mongo?.closeAll(); await pools?.close(); }); + +describe('solo mode', () => { + let app; + beforeAll(async () => { ({ app } = await build({})); }); + + it('describes the lab', async () => { + expect((await request(app).get('/api/config')).body).toEqual({ mode: 'solo', eventCodeRequired: false }); + const r = await request(app).get('/api/patterns'); + expect(r.status).toBe(200); + expect(r.body).toHaveLength(6); + expect(r.body[0].cards.document[0]).toHaveProperty('sql'); + }); + it('runs SQL and returns Oracle errors with HTTP 200', async () => { + const ok = await request(app).post('/api/run').send({ lane: 'sql', patternId: null, text: 'SELECT 1 AS n FROM dual' }); + expect(ok.status).toBe(200); + expect(ok.body.results[0].rows[0].N).toBe(1); + const bad = await request(app).post('/api/run').send({ lane: 'sql', patternId: null, text: 'SELECT * FROM no_such_table_xyz' }); + expect(bad.status).toBe(200); + expect(bad.body.results[0].code).toBe('ORA-00942'); + }); + it('runs a MongoDB command', async () => { + const r = await request(app).post('/api/run').send({ lane: 'mongo', patternId: null, text: 'show collections' }); + expect(r.status).toBe(200); + expect(r.body.results[0].kind).toBe('collections'); + }); + it('rejects a bad lane with 400', async () => { + expect((await request(app).post('/api/run').send({ lane: 'js', text: 'x' })).status).toBe(400); + }); + it('maps a paused gate to 423', async () => { + const { app: a, gate } = await build({}); + gate.pause(true); + const r = await request(a).post('/api/run').send({ lane: 'sql', patternId: null, text: 'SELECT 1 FROM dual' }); + expect(r.status).toBe(423); + expect(r.body.error).toBe('paused by instructor'); + }); + it('maps queue timeout to 429', async () => { + const { app: a, gate } = await build({ QUEUE_TIMEOUT_MS: '200' }); + gate.run({ userId: 'someone-else', label: 'hold' }, () => new Promise((res) => setTimeout(res, 1500))); + const r = await request(a).post('/api/run').send({ lane: 'sql', patternId: null, text: 'SELECT 1 FROM dual' }); + expect(r.status).toBe(429); + expect(r.body.error).toBe('busy, retry'); + }); + it('reports the queue position', async () => { + const r = await request(app).get('/api/queue'); + expect(r.body).toEqual({ position: null, depth: 0, paused: false }); + }); +}); + +describe('event mode', () => { + let app; let workspaces; + beforeAll(async () => { ({ app, workspaces } = await build({ LAB_MODE: 'event', ADMIN_PASSWORD: 'admin-test', EVENT_CODE: 'LAB26' })); }); + // Only the workspace(s) this block signed in: never another run's or a live event's. + afterAll(async () => { + const mine = (await Promise.all(['a@example.com'].map((e) => workspaces.findByEmail(e)))).filter(Boolean); + await dropOwn(workspaces, mongo, mine.map((w) => w.schema)); + }, 300000); + + it('requires sign-in', async () => { + expect((await request(app).get('/api/me')).status).toBe(401); + expect((await request(app).post('/api/run').send({ lane: 'sql', text: 'SELECT 1 FROM dual' })).status).toBe(401); + }); + it('checks the event code', async () => { + const r = await request(app).post('/api/signin').send({ name: 'A', email: 'a@example.com', code: 'nope' }); + expect(r.status).toBe(403); + }); + it('signs in, then runs SQL in the attendee workspace', async () => { + const agent = request.agent(app); + const s = await agent.post('/api/signin').send({ name: 'A', email: 'a@example.com', code: 'LAB26' }); + expect(s.status).toBe(200); + expect(s.body.schema).toMatch(/^WS_/); + const me = await agent.get('/api/me'); + expect(me.body.user.email).toBe('a@example.com'); + const r = await agent.post('/api/run').send({ lane: 'sql', patternId: null, text: 'SELECT USER AS u FROM dual' }); + expect(r.body.results[0].rows[0].U).toBe(s.body.schema); + }, 120000); + it('protects the admin API', async () => { + expect((await request(app).get('/api/admin/status')).status).toBe(401); + expect((await request(app).post('/api/admin/login').send({ password: 'wrong' })).status).toBe(403); + const admin = request.agent(app); + expect((await admin.post('/api/admin/login').send({ password: 'admin-test' })).status).toBe(200); + const st = await admin.get('/api/admin/status'); + expect(st.status).toBe(200); + expect(st.body.gate.permits).toBe(1); + expect(st.body.attendees.length).toBeGreaterThanOrEqual(1); + expect((await admin.post('/api/admin/pause').send({ on: true })).body).toEqual({ paused: true }); + expect((await admin.post('/api/admin/pause').send({ on: false })).body).toEqual({ paused: false }); + expect((await admin.post('/api/admin/timeouts').send({ sqlMs: 5000, mongoMs: 5000 })).body).toEqual({ sqlMs: 5000, mongoMs: 5000 }); + expect((await admin.post('/api/admin/timeouts').send({ sqlMs: 5, mongoMs: 5000 })).status).toBe(400); + }); + it('resets every attendee in the background (202) and reports progress in status', async (ctx) => { + const a = await workspaces.findByEmail('a@example.com'); + // Reset-all rebuilds EVERY workspace: only run when the only one is this test's own. + if (await skipUnlessOnlyOwn(ctx, workspaces, [a.schema])) return; + const admin = request.agent(app); + expect((await admin.post('/api/admin/login').send({ password: 'admin-test' })).status).toBe(200); + const r = await admin.post('/api/admin/reset-attendee').send({ schema: '*' }); + expect(r.status).toBe(202); + expect(r.body).toEqual({ started: 1 }); + let st; + const deadline = Date.now() + 240000; + do { + await new Promise((res) => { setTimeout(res, 500); }); + st = (await admin.get('/api/admin/status')).body; + } while (st.resets.running && Date.now() < deadline); + expect(st.resets).toMatchObject({ running: false, total: 1, done: 1, failed: [] }); + expect(st.pending).toBe(0); + expect(st.attendees.find((x) => x.schema === a.schema).built).toHaveLength(6); + expect((await admin.post('/api/admin/reset-attendee').send({ schema: a.schema })).body).toEqual({ ok: true, reset: 1 }); + }, 300000); +}); diff --git a/labs/converged-modeling-patterns/app/test/integration/mongo.test.js b/labs/converged-modeling-patterns/app/test/integration/mongo.test.js new file mode 100644 index 0000000..96961c7 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/integration/mongo.test.js @@ -0,0 +1,66 @@ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { testConfig } from './env.js'; +import { bootstrap, createPools, executeSql } from '../../src/db/oracle.js'; +import { MongoPool, runMongo, mongoUri } from '../../src/db/mongo.js'; +import { parseMongoCommand } from '../../src/content/mongoCommand.js'; + +const cfg = testConfig(); +const sqlOpts = { timeoutMs: 10000, maxRows: 500, maxBytes: 1048576 }; +const mOpts = { timeoutMs: 10000, maxRows: 500, maxBytes: 1048576 }; +let pools; let conn; let mongo; let client; +const run = (text) => runMongo(client, 'CMP_USER', parseMongoCommand(text), mOpts); + +beforeAll(async () => { + await bootstrap(cfg); + pools = await createPools(cfg); + conn = await pools.exec.getConnection({ user: cfg.db.cmpUser, password: cfg.db.cmpPassword }); + await executeSql(conn, 'DROP TABLE lab_it_docs PURGE', sqlOpts); + await executeSql(conn, 'CREATE JSON COLLECTION TABLE lab_it_docs', sqlOpts); + mongo = new MongoPool(cfg); + client = mongo.client(cfg.db.cmpUser, cfg.db.cmpPassword); +}); +afterAll(async () => { + await mongo?.closeAll(); + await executeSql(conn, 'DROP TABLE lab_it_docs PURGE', sqlOpts); + await conn?.close(); + await pools?.close(); +}); + +describe('mongo layer', () => { + it('builds the same URI shape run.sh uses', () => { + expect(mongoUri({ host: 'oracle', port: 27017, user: 'WS_ABC', password: 'p@ss' })) + .toBe('mongodb://WS_ABC:p%40ss@oracle:27017/WS_ABC?authMechanism=PLAIN&authSource=$external&retryWrites=false&loadBalanced=true'); + }); + it('inserts, finds and counts', async () => { + expect(await run('db.lab_it_docs.insertMany([{_id:"a", n:1},{_id:"b", n:2},{_id:"c", n:3}])')).toMatchObject({ kind: 'write' }); + const r = await run('db.lab_it_docs.find({n:{$gte:2}}).sort({n:-1})'); + expect(r.kind).toBe('docs'); + expect(r.docs.map((d) => d._id)).toEqual(['c', 'b']); + expect(await run('db.lab_it_docs.countDocuments({})')).toMatchObject({ kind: 'count', count: 3 }); + }); + it('updates and reads back with findOne', async () => { + await run('db.lab_it_docs.updateOne({_id:"a"}, {$set:{n:10}})'); + const r = await run('db.lab_it_docs.findOne({_id:"a"})'); + expect(r.docs[0].n).toBe(10); + }); + it('runs $sql in an aggregation pipeline over the wire', async () => { + const r = await run('db.aggregate([{ $sql: `select 1 as "one" from dual` }])'); + expect(r.kind).toBe('docs'); + expect(r.docs[0].one).toBe(1); + }); + it('lists collections', async () => { + const r = await run('show collections'); + expect(r.kind).toBe('collections'); + expect(r.names.map((n) => n.toLowerCase())).toContain('lab_it_docs'); + }); + it('returns errors instead of throwing', async () => { + const r = await run('db.aggregate([{ $sql: `select * from no_such_table_xyz` }])'); + expect(r.kind).toBe('error'); + expect(r.error).toBeTruthy(); + }); + it('caps documents at maxRows', async () => { + const r = await runMongo(client, 'CMP_USER', parseMongoCommand('db.lab_it_docs.find({})'), { ...mOpts, maxRows: 2 }); + expect(r.docs).toHaveLength(2); + expect(r.truncated).toBe(true); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/integration/oracle.test.js b/labs/converged-modeling-patterns/app/test/integration/oracle.test.js new file mode 100644 index 0000000..11e09c8 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/integration/oracle.test.js @@ -0,0 +1,68 @@ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { testConfig } from './env.js'; +import { bootstrap, createPools, executeSql, readStats, STAT_NAMES } from '../../src/db/oracle.js'; + +const cfg = testConfig(); +const opts = { timeoutMs: 10000, maxRows: 500, maxBytes: 1048576 }; +let pools; let conn; + +beforeAll(async () => { + await bootstrap(cfg); + await bootstrap(cfg); // idempotent + pools = await createPools(cfg); + conn = await pools.exec.getConnection({ user: cfg.db.cmpUser, password: cfg.db.cmpPassword }); + await executeSql(conn, 'DROP TABLE lab_it_t PURGE', opts); + await executeSql(conn, 'CREATE TABLE lab_it_t (id NUMBER PRIMARY KEY, v VARCHAR2(20))', opts); +}); +afterAll(async () => { + await executeSql(conn, 'DROP TABLE lab_it_t PURGE', opts); + await conn?.close(); + await pools?.close(); +}); + +describe('oracle layer', () => { + it('returns rows with column names', async () => { + const r = await executeSql(conn, "SELECT 1 AS n, 'a' AS s FROM dual", opts); + expect(r).toMatchObject({ kind: 'rows', columns: ['N', 'S'], rows: [{ N: 1, S: 'a' }], rowCount: 1, truncated: false }); + }); + it('returns rowsAffected for DML', async () => { + expect(await executeSql(conn, "INSERT INTO lab_it_t VALUES (1, 'x')", opts)).toMatchObject({ kind: 'dml', rowsAffected: 1 }); + }); + it('returns ok for DDL and PL/SQL', async () => { + expect(await executeSql(conn, 'BEGIN NULL; END;', opts)).toMatchObject({ kind: 'ok' }); + }); + it('returns Oracle errors with their code instead of throwing', async () => { + const r = await executeSql(conn, 'SELECT * FROM no_such_table_xyz', opts); + expect(r).toMatchObject({ kind: 'error', code: 'ORA-00942' }); + expect(r.error).toMatch(/ORA-00942/); + }); + it('caps rows at maxRows and flags truncation', async () => { + const r = await executeSql(conn, 'SELECT level AS n FROM dual CONNECT BY level <= 600', opts); + expect(r.rowCount).toBe(500); + expect(r.truncated).toBe(true); + }); + it('caps results at maxBytes and flags truncation', async () => { + const r = await executeSql(conn, "SELECT RPAD('x', 4000, 'x') AS s FROM dual CONNECT BY level <= 400", opts); + expect(JSON.stringify(r.rows).length).toBeLessThanOrEqual(1048576); + expect(r.truncated).toBe(true); + }); + it('times out a long call', async () => { + const r = await executeSql(conn, 'BEGIN DBMS_SESSION.SLEEP(5); END;', { ...opts, timeoutMs: 500 }); + expect(r.kind).toBe('error'); + }); + it('reads session statistics that grow with a write', async () => { + const before = await readStats(conn); + expect(Object.keys(before).sort()).toEqual([...STAT_NAMES].sort()); + await executeSql(conn, "UPDATE lab_it_t SET v = 'y' WHERE id = 1", { ...opts, autoCommit: false }); + const after = await readStats(conn); + await conn.rollback(); + expect(after['redo size']).toBeGreaterThan(before['redo size']); + expect(after['db block changes']).toBeGreaterThan(before['db block changes']); + }); + it('control pool connects as LAB_ADMIN', async () => { + const c = await pools.control.getConnection(); + const r = await executeSql(c, 'SELECT USER AS u FROM dual', opts); + await c.close(); + expect(r.rows[0].U).toBe('LAB_ADMIN'); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/integration/runner.test.js b/labs/converged-modeling-patterns/app/test/integration/runner.test.js new file mode 100644 index 0000000..d105217 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/integration/runner.test.js @@ -0,0 +1,241 @@ +// app/test/integration/runner.test.js +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import path from 'node:path'; +import { testConfig } from './env.js'; +import { bootstrap, createPools } from '../../src/db/oracle.js'; +import { MongoPool } from '../../src/db/mongo.js'; +import { Gate } from '../../src/gate.js'; +import { ResultCache } from '../../src/cache.js'; +import { loadPatterns } from '../../src/content/patterns.js'; +import { Workspaces } from '../../src/services/workspaces.js'; +import { Runner } from '../../src/services/runner.js'; + +const cfg = testConfig(); +const patterns = loadPatterns(path.resolve(import.meta.dirname, '../../../patterns')); +let pools; let mongo; let workspaces; let runner; let user; + +async function makeRunner() { + workspaces = new Workspaces({ cfg, pools }); + await workspaces.load(); + return new Runner({ cfg, gate: new Gate(cfg.gate), cache: new ResultCache(cfg.cache), pools, mongo, workspaces, patterns }); +} + +beforeAll(async () => { + await bootstrap(cfg); + pools = await createPools(cfg); + mongo = new MongoPool(cfg); + runner = await makeRunner(); + user = { id: 'solo', workspace: workspaces.solo() }; +}, 300000); +afterAll(async () => { await mongo?.closeAll(); await pools?.close(); }); + +describe.each(patterns.map((p) => [p.id, p]))('%s', (id, p) => { + it('resets cleanly and every step runs in order', async () => { + const r = await runner.reset({ user, patternId: id }); + expect(r.errors, JSON.stringify(r.errors)).toEqual([]); + for (const lane of ['document', 'converged']) { + for (const st of p.lanes[lane]) { + const out = await runner.runSql({ user, patternId: id, text: st.sql }); + const errs = out.results.filter((x) => x.kind === 'error'); + expect(errs, `${lane} step "${st.title}" (line ${st.line})`).toEqual([]); + } + } + }, 120000); + + it('runs every MongoDB card after a reset', async () => { + await runner.reset({ user, patternId: id }); + for (const card of p.lanes.mongo) { + const out = await runner.runMongoText({ user, patternId: id, text: card.command }); + expect(out.results[0].kind, card.title).not.toBe('error'); + if (out.results[0].kind === 'docs') expect(out.results[0].docs.length, card.title).toBeGreaterThan(0); + } + }, 120000); + + it('measure-it: the document model writes more than the converged model', async () => { + await runner.reset({ user, patternId: id }); + for (const m of p.measures) { + const r = await runner.measure({ user, patternId: id, tag: m.tag }); + expect(r.document.result.kind, m.tag).not.toBe('error'); + expect(r.converged.result.kind, m.tag).not.toBe('error'); + // Directional claim of the lecture. If this fails, STOP and report both stat sets — + // do not relax the assertion (see the plan's note on OSON partial updates). + expect(r.document.stats['redo size'], `${m.tag} redo`).toBeGreaterThan(r.converged.stats['redo size']); + expect(r.document.stats['db block changes'], `${m.tag} blocks`).toBeGreaterThan(r.converged.stats['db block changes']); + } + }, 120000); + + it('measure-it: write stats are non-zero and stable across repeated runs', async () => { + await runner.reset({ user, patternId: id }); + for (const m of p.measures) { + const runs = []; + for (let i = 0; i < 3; i++) runs.push(await runner.measure({ user, patternId: id, tag: m.tag })); + for (const side of ['document', 'converged']) { + const pick = (r) => ({ redo: r[side].stats['redo size'], blocks: r[side].stats['db block changes'] }); + for (const r of runs) { + // 'ok' = a PL/SQL block (03-bucket's document side), which always writes. + expect(['dml', 'ok'], `${m.tag} ${side}`).toContain(r[side].result.kind); + if (r[side].result.kind === 'ok' || r[side].result.rowsAffected > 0) { + expect(r[side].stats['redo size'], `${m.tag} ${side} redo`).toBeGreaterThan(0); + expect(r[side].stats['db block changes'], `${m.tag} ${side} blocks`).toBeGreaterThan(0); + } + } + // Single-row INSERTs flip by +76 redo / +1 block run to run (space management + // rolled back and redone), and a whole-document LOB rewrite occasionally steps + // up when the LOB segment grows (seen: 21568/29 vs 20196/15). So stability is + // modal agreement: at least 2 of the 3 runs agree within 100 redo and 1 block. + const agree = (a, b) => Math.abs(a.redo - b.redo) <= 100 && Math.abs(a.blocks - b.blocks) <= 1; + const picks = runs.map(pick); + const modal = picks.some((a, i) => picks.some((b, j) => i !== j && agree(a, b))); + expect(modal, `${m.tag} ${side}: no 2 of 3 runs agree ${JSON.stringify(picks)}`).toBe(true); + } + // The lecture's direction holds on every run, not just on average. + for (const r of runs) { + expect(r.document.stats['redo size'], `${m.tag} run redo`).toBeGreaterThan(r.converged.stats['redo size']); + expect(r.document.stats['db block changes'], `${m.tag} run blocks`).toBeGreaterThan(r.converged.stats['db block changes']); + } + } + }, 180000); +}); + +describe('cache and dirty flags', () => { + const id = '01-extended-reference'; + const readCard = () => patterns.find((p) => p.id === id).lanes.converged.find((s) => /^\s*(SELECT|WITH)\b/i.test(s.sql)); + + it('serves a repeated read from cache and stops after a write', async () => { + await runner.reset({ user, patternId: id }); + const text = readCard().sql; + expect((await runner.runSql({ user, patternId: id, text })).cached).toBe(false); + expect((await runner.runSql({ user, patternId: id, text })).cached).toBe(true); + await runner.runSql({ user, patternId: id, text: "UPDATE xr_advisors SET office = 'LAB-TEST' WHERE ROWNUM = 1" }); + expect((await runner.runSql({ user, patternId: id, text })).cached).toBe(false); + }); + + it('keeps a dirty workspace uncached after an app restart', async () => { + runner = await makeRunner(); // fresh cache, dirty flags reloaded from the database + const text = readCard().sql; + expect((await runner.runSql({ user, patternId: id, text })).cached).toBe(false); + expect((await runner.runSql({ user, patternId: id, text })).cached).toBe(false); + await runner.reset({ user, patternId: id }); + await runner.runSql({ user, patternId: id, text }); + expect((await runner.runSql({ user, patternId: id, text })).cached).toBe(true); + }); +}); + +describe('results', () => { + it('returns Oracle errors as results', async () => { + const out = await runner.runSql({ user, patternId: null, text: 'SELECT * FROM no_such_table_xyz' }); + expect(out.results[0]).toMatchObject({ kind: 'error', code: 'ORA-00942' }); + }); + it('truncates results over 1 MB', async () => { + const out = await runner.runSql({ user, patternId: null, text: "SELECT RPAD('x', 4000, 'x') AS s FROM dual CONNECT BY level <= 400" }); + expect(out.results[0].truncated).toBe(true); + }); + it('blocks guarded statements without executing them', async () => { + const out = await runner.runSql({ user, patternId: null, text: 'GRANT DBA TO cmp_user' }); + expect(out.results[0]).toMatchObject({ kind: 'error', code: 'LAB-GUARD' }); + }); + it('returns Mongo parse errors as results', async () => { + const out = await runner.runMongoText({ user, patternId: null, text: 'db.x.drop()' }); + expect(out.results[0]).toMatchObject({ kind: 'error', code: 'LAB-PARSE' }); + }); + it('runs several statements in one request', async () => { + const out = await runner.runSql({ user, patternId: null, text: 'SELECT 1 FROM dual;\nSELECT 2 FROM dual;' }); + expect(out.results).toHaveLength(2); + }); +}); + +describe('mongo timeout (addendum A)', () => { + // The addendum's `connect by level <= 20000000` fails fast here with ORA-30009 (CONNECT BY memory), + // so use a deterministic ~5 s cross join of three small CONNECT BY row sources instead. + const SLOW = 'db.aggregate([{ $sql: "select count(*) as n from (select level l from dual connect by level <= 5000) a, ' + + '(select level l from dual connect by level <= 5000) b, (select level l from dual connect by level <= 10) c" }])'; + + it('stops a slow MongoDB op at the timeout, holds the queue until the session is idle, and serves the next request', async () => { + const slowCfg = testConfig({ MONGO_TIMEOUT_MS: '1000' }); + const slowRunner = new Runner({ + cfg: slowCfg, gate: new Gate(slowCfg.gate), cache: new ResultCache(slowCfg.cache), pools, mongo, workspaces, patterns, + }); + const schema = user.workspace.schema.toUpperCase(); + // Each sample: the executing statements of ACTIVE CMP_USER sessions, as "sid:sql_id:sql_exec_id". + const activeOps = async () => { + const c = await pools.control.getConnection(); + try { + const r = await c.execute( + "SELECT sid, sql_id, sql_exec_id FROM v$session WHERE username = :s AND status = 'ACTIVE' AND sql_id IS NOT NULL", + { s: schema }, + ); + return r.rows.map(([sid, sqlId, execId]) => `${sid}:${sqlId}:${execId}`); + } finally { + await c.close(); + } + }; + const samples = []; + let sampling = true; + const sampler = (async () => { + while (sampling) { + samples.push(await activeOps()); + await new Promise((res) => { setTimeout(res, 100); }); + } + })(); + const started = Date.now(); + const out = await slowRunner.runMongoText({ user, patternId: null, text: SLOW }); + const elapsed = Date.now() - started; + const atReturn = await activeOps(); + sampling = false; + await sampler; + + expect(out.results[0], JSON.stringify(out.results[0])).toMatchObject({ kind: 'error', code: 'LAB-TIMEOUT' }); + expect(out.results[0].error).toBe('MongoDB operation exceeded 1 s and was stopped'); + expect(elapsed).toBeGreaterThanOrEqual(1000); + // An attendee operation persists across samples. When the client is dropped, ORDS briefly runs its + // own housekeeping on a second pooled session (seen for a single sample, under the same username); + // that is not an attendee operation, so only operations seen in >= 2 samples are counted. + const seen = new Map(); + samples.flat().forEach((op) => seen.set(op, (seen.get(op) ?? 0) + 1)); + const ops = [...seen].filter(([, n]) => n >= 2).map(([op]) => op); + expect(ops.length, JSON.stringify(samples)).toBeGreaterThanOrEqual(1); // the sampler did see the slow op + const maxConcurrent = Math.max(...samples.map((s) => s.filter((op) => ops.includes(op)).length)); + expect(maxConcurrent, JSON.stringify(samples)).toBeLessThanOrEqual(1); + // The permit was held until the database stopped executing the timed-out op. + expect(atReturn.filter((op) => ops.includes(op))).toEqual([]); + const next = await slowRunner.runSql({ user, patternId: null, text: 'SELECT 1 FROM dual' }); + expect(next.results[0].kind).toBe('rows'); + }, 120000); +}); + +describe('run budget and cancel (I3)', () => { + const SLEEP = 'BEGIN DBMS_SESSION.SLEEP(1); END;\n/'; + const shortRunner = (env) => { + const c = testConfig(env); + const gate = new Gate(c.gate); + return { gate, r: new Runner({ cfg: c, gate, cache: new ResultCache(c.cache), pools, mongo, workspaces, patterns }) }; + }; + + it('gives a multi-statement run one deadline and stops it with LAB-TIMEOUT', async () => { + const { r } = shortRunner({ SQL_TIMEOUT_MS: '1500' }); + const started = Date.now(); + const out = await r.runSql({ user, patternId: null, text: [SLEEP, SLEEP, SLEEP, 'SELECT 1 FROM dual;'].join('\n') }); + const elapsed = Date.now() - started; + expect(out.results).toHaveLength(4); + expect(out.results[0].kind, JSON.stringify(out.results[0])).toBe('ok'); + // The second sleep gets only the ~0.5 s left and is stopped; the rest never run. + expect(out.results.slice(1).map((x) => x.code), JSON.stringify(out.results)).toEqual(['LAB-TIMEOUT', 'LAB-TIMEOUT', 'LAB-TIMEOUT']); + expect(elapsed).toBeLessThan(3000); + const next = await r.runSql({ user, patternId: null, text: 'SELECT 1 FROM dual' }); + expect(next.results[0].kind).toBe('rows'); + }, 60000); + + it('cancel mid-loop stops the remaining statements', async () => { + const { r, gate } = shortRunner({ SQL_TIMEOUT_MS: '20000' }); + const started = Date.now(); + const run = r.runSql({ user, patternId: null, text: [SLEEP, SLEEP, SLEEP, SLEEP].join('\n') }); + await new Promise((res) => { setTimeout(res, 400); }); + const [running] = gate.status().running; + expect(gate.cancel(running.id)).toBe(true); + const out = await run; + const elapsed = Date.now() - started; + expect(out.results).toHaveLength(4); + expect(out.results.slice(1).map((x) => x.code), JSON.stringify(out.results)).toEqual(['LAB-CANCELLED', 'LAB-CANCELLED', 'LAB-CANCELLED']); + expect(elapsed).toBeLessThan(2500); + }, 60000); +}); diff --git a/labs/converged-modeling-patterns/app/test/integration/workspaces.test.js b/labs/converged-modeling-patterns/app/test/integration/workspaces.test.js new file mode 100644 index 0000000..a197143 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/integration/workspaces.test.js @@ -0,0 +1,220 @@ +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; +import path from 'node:path'; +import { setTimeout as sleep } from 'node:timers/promises'; +import oracledb from 'oracledb'; +import { testConfig, dropOwn, skipUnlessOnlyOwn } from './env.js'; +import { bootstrap, createPools, executeSql } from '../../src/db/oracle.js'; +import { MongoPool } from '../../src/db/mongo.js'; +import { Gate } from '../../src/gate.js'; +import { ResultCache } from '../../src/cache.js'; +import { loadPatterns } from '../../src/content/patterns.js'; +import { Workspaces } from '../../src/services/workspaces.js'; +import { Runner } from '../../src/services/runner.js'; + +const cfg = testConfig({ LAB_MODE: 'event', ADMIN_PASSWORD: 'admin-test' }); +const connectString = `${cfg.db.host}:${cfg.db.port}/${cfg.db.service}`; +const patterns = loadPatterns(path.resolve(import.meta.dirname, '../../../patterns')); +let pools; let mongo; let ws; let runner; +// Every workspace this file creates. Cleanup touches these and nothing else. +const own = new Set(); +const mine = (w) => { own.add(w.schema); return w; }; +const EMAILS = ['a', 'b', 'c', 'held', 'fresh', 'reaper', 'stuck'].map((x) => `${x}@example.com`); + +// Park + reap our own workspaces. One still held by an ORDS session gets the reaper's kill +// path at once (its parked time is backdated — only for our own rows) instead of in 2 min. +async function cleanupOwn(schemas) { + const r = await dropOwn(ws, mongo, schemas); + if (!r.pending) return r; + await control("UPDATE lab_pending_drop SET requested_at = SYSTIMESTAMP - INTERVAL '3' MINUTE " + + `WHERE schema_name IN (${schemas.map((_, i) => `:s${i}`).join(', ')})`, + Object.fromEntries(schemas.map((x, i) => [`s${i}`, x]))); + return ws.reapPending(schemas); +} + +// A held connection is a deterministic stand-in for an ORDS-side session that outlives its +// client (see task-11-report.md): opening one directly against a workspace guarantees +// DROP USER ... CASCADE hits ORA-01940 until it's closed, with no dependence on ORDS timing. +async function openAs(workspace) { + return oracledb.getConnection({ user: workspace.schema, password: workspace.password, connectString }); +} + +async function countWhere(sql) { + const c = await pools.control.getConnection(); + try { + const r = await executeSql(c, sql, { timeoutMs: 10000, maxRows: 5, maxBytes: 10000 }); + return r.rows[0].N; + } finally { + await c.close(); + } +} + +async function control(sql, binds = {}) { + const c = await pools.control.getConnection(); + try { + return await c.execute(sql, binds, { autoCommit: true, outFormat: oracledb.OUT_FORMAT_OBJECT }); + } finally { + await c.close(); + } +} + +beforeAll(async () => { + await bootstrap(cfg); + pools = await createPools(cfg); + mongo = new MongoPool(cfg); + ws = new Workspaces({ cfg, pools }); + await ws.load(); + // Clean slate for this file only: workspaces an earlier, interrupted run left behind under + // this file's own test emails. Nobody else's workspace is touched. + const leftovers = (await Promise.all(EMAILS.map((e) => ws.findByEmail(e)))).filter(Boolean).map((w) => w.schema); + if (leftovers.length) await cleanupOwn(leftovers); + runner = new Runner({ cfg, gate: new Gate(cfg.gate), cache: new ResultCache(cfg.cache), pools, mongo, workspaces: ws, patterns }); +}, 300000); +afterAll(async () => { + if (ws) await cleanupOwn([...own]); + await mongo?.closeAll(); await pools?.close(); +}, 300000); + +describe('event workspaces', () => { + it('keeps a stable session secret', async () => { + const a = await ws.sessionSecret(); + expect(a).toMatch(/^[0-9a-f]{64}$/); + expect(await ws.sessionSecret()).toBe(a); + }); + + it('assigns one workspace per email and returns it again for the same email', async () => { + const a = mine(await ws.assign({ email: 'a@example.com', name: 'A' })); + const b = mine(await ws.assign({ email: 'b@example.com', name: 'B' })); + expect(a.schema).toMatch(/^WS_[0-9A-F]{6}$/); + expect(b.schema).not.toBe(a.schema); + expect((await ws.assign({ email: 'a@example.com', name: 'A again' })).schema).toBe(a.schema); + expect((await ws.findBySchema(a.schema)).email).toBe('a@example.com'); + }, 120000); + + it('isolates attendees from each other', async () => { + const a = await ws.findByEmail('a@example.com'); + const b = await ws.findByEmail('b@example.com'); + const userA = { id: a.schema, workspace: a }; + const userB = { id: b.schema, workspace: b }; + expect((await runner.reset({ user: userB, patternId: '01-extended-reference' })).ok).toBe(true); + const out = await runner.runSql({ user: userA, patternId: null, text: `SELECT COUNT(*) FROM ${b.schema}.xr_advisors` }); + expect(out.results[0].kind).toBe('error'); + expect(['ORA-00942', 'ORA-01031']).toContain(out.results[0].code); + }, 180000); + + it('serves the MongoDB API per workspace', async () => { + const a = await ws.findByEmail('a@example.com'); + const user = { id: a.schema, workspace: a }; + const p = patterns.find((x) => x.id === '01-extended-reference'); + const out = await runner.runMongoText({ user, patternId: p.id, text: p.lanes.mongo[0].command }); + expect(out.results[0].kind).toBe('docs'); + expect(out.results[0].docs.length).toBeGreaterThan(0); + }, 180000); + + it('claims a prewarmed workspace before creating a new one', async (ctx) => { + // Another prewarmed workspace could be claimed instead of ours: only run on a clean DB. + if (await skipUnlessOnlyOwn(ctx, ws, own)) return; + const pre = mine(await ws.provision()); + const before = (await ws.list()).length; + const c = mine(await ws.assign({ email: 'c@example.com', name: 'C' })); + expect(c.schema).toBe(pre.schema); + expect((await ws.list()).length).toBe(before); + }, 120000); + + it('end event locks and parks every workspace and returns; reapPending drops the free one and finishes the held one once its connection closes', async (ctx) => { + // dropAll() drops EVERY workspace: only run when all of them are this file's own. + if (await skipUnlessOnlyOwn(ctx, ws, own)) return; + // Deterministic stand-in for an ORDS-held session: this connection guarantees `held` + // hits ORA-01940 on DROP USER for as long as it stays open — no dependence on ORDS + // session-release timing (see task-11-report.md's evidence that timing is unpredictable + // enough to make the previous version of this test flaky). + const held = mine(await ws.provision({ email: 'held@example.com', name: 'Held' })); + const heldConn = await openAs(held); + + // No connection ever opened against this one, so the first reap pass must drop it. + const fresh = mine(await ws.provision({ email: 'fresh@example.com', name: 'Fresh' })); + + const parked = await ws.dropAll(mongo); + expect(parked.dropped).toBe(0); // dropAll only locks + parks; the reaper drops + expect(parked.pending).toBeGreaterThanOrEqual(2); + expect(await ws.list()).toEqual([]); // pending workspaces are invisible to list() + + const { dropped, pending } = await ws.reapPending([...own]); + expect(dropped).toBeGreaterThanOrEqual(1); // at least `fresh` + expect(pending).toBeGreaterThanOrEqual(1); // at least `held` + + expect(await countWhere(`SELECT COUNT(*) AS n FROM all_users WHERE username = '${fresh.schema}'`)).toBe(0); + expect(await countWhere(`SELECT COUNT(*) AS n FROM lab_pending_drop WHERE schema_name = '${held.schema}'`)).toBe(1); + + // Locked immediately — no new connection succeeds, regardless of the still-open old one. + await expect(openAs(held)).rejects.toThrow(/ORA-28000/); + + // Once the connection that was blocking it closes, reapPending() finishes the job. + await heldConn.close(); + await ws.reapPending([held.schema]); + expect(await countWhere(`SELECT COUNT(*) AS n FROM lab_pending_drop WHERE schema_name = '${held.schema}'`)).toBe(0); + expect(await countWhere(`SELECT COUNT(*) AS n FROM all_users WHERE username = '${held.schema}'`)).toBe(0); + }, 60000); + + it('startReaper drops a parked workspace on its own once unblocked, and stops cleanly', async (ctx) => { + // dropAll() and the unfiltered reaper touch every workspace: only run on our own rows. + if (await skipUnlessOnlyOwn(ctx, ws, own)) return; + const held = mine(await ws.provision({ email: 'reaper@example.com', name: 'Reaper' })); + const heldConn = await openAs(held); + await ws.dropAll(mongo); // locks + parks `held`; the open connection guarantees ORA-01940 + + const spy = vi.spyOn(ws, 'reapPending'); + const stop = ws.startReaper(500); + try { + await heldConn.close(); // now droppable — the reaper's own next tick should pick it up + + const deadline = Date.now() + 20000; + let gone = false; + while (Date.now() < deadline) { + if (await countWhere(`SELECT COUNT(*) AS n FROM lab_pending_drop WHERE schema_name = '${held.schema}'`) === 0) { + gone = true; + break; + } + await sleep(300); + } + expect(gone).toBe(true); + } finally { + stop(); + } + + // No further passes run once stopped. + const callsAtStop = spy.mock.calls.length; + await sleep(1500); // several more 500ms intervals' worth + expect(spy.mock.calls.length).toBe(callsAtStop); + spy.mockRestore(); + + expect(await countWhere(`SELECT COUNT(*) AS n FROM all_users WHERE username = '${held.schema}'`)).toBe(0); + }, 30000); + + it('reapPending kills the sessions of a workspace parked 2+ min, drops it, and skips a non-WS_ row without stopping', async (ctx) => { + if (await skipUnlessOnlyOwn(ctx, ws, own)) return; + const held = mine(await ws.provision({ email: 'stuck@example.com', name: 'Stuck' })); + const heldConn = await openAs(held); // stands in for an ORDS pool session that never lets go + await ws.dropAll(mongo); // parks `held`: parked < 2 min, so no kill yet + expect(await countWhere(`SELECT COUNT(*) AS n FROM lab_pending_drop WHERE schema_name = '${held.schema}'`)).toBe(1); + + await control("MERGE INTO lab_pending_drop d USING (SELECT 'NOT_A_WS' AS s FROM dual) x ON (d.schema_name = x.s) " + + 'WHEN NOT MATCHED THEN INSERT (schema_name) VALUES (x.s)'); + // Backdate only the rows this test parked — never a real workspace on the shared DB. + await control("UPDATE lab_pending_drop SET requested_at = SYSTIMESTAMP - INTERVAL '3' MINUTE WHERE schema_name IN (:held, 'NOT_A_WS')", + { held: held.schema }); + try { + await ws.reapPending([held.schema, 'NOT_A_WS']); + + expect(await countWhere(`SELECT COUNT(*) AS n FROM all_users WHERE username = '${held.schema}'`)).toBe(0); + expect(await countWhere(`SELECT COUNT(*) AS n FROM lab_pending_drop WHERE schema_name = '${held.schema}'`)).toBe(0); + await expect(heldConn.execute('SELECT 1 FROM dual')).rejects.toThrow(); + + const bogus = (await control("SELECT last_error FROM lab_pending_drop WHERE schema_name = 'NOT_A_WS'")).rows; + expect(bogus).toHaveLength(1); + expect(bogus[0].LAST_ERROR).toMatch(/not a WS_ workspace name/); + } finally { + await heldConn.close().catch(() => {}); + await control("DELETE FROM lab_pending_drop WHERE schema_name = 'NOT_A_WS'"); + } + }, 60000); +}); diff --git a/labs/converged-modeling-patterns/app/test/load/load.mjs b/labs/converged-modeling-patterns/app/test/load/load.mjs new file mode 100644 index 0000000..5a24bc1 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/load/load.mjs @@ -0,0 +1,118 @@ +// app/test/load/load.mjs +// Requires LAB_MODE=event. Env: LAB_URL, EVENT_CODE, USERS (150), WRITERS (0.2), +// DB_HOST/DB_PORT (host-mapped, default localhost:1522), LAB_ADMIN_PASSWORD, +// ADMIN_PASSWORD (to prewarm workspaces before sign-in; PREWARM=0 skips it). +import oracledb from 'oracledb'; + +const URL = process.env.LAB_URL ?? 'http://localhost:3100'; +const USERS = Number(process.env.USERS ?? 150); +const WRITERS = Number(process.env.WRITERS ?? 0.2); +const PATTERN = '01-extended-reference'; +const pct = (a, p) => a.sort((x, y) => x - y)[Math.min(a.length - 1, Math.floor(p * a.length))] ?? 0; +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + +async function post(jar, path, body) { + const res = await fetch(URL + path, { method: 'POST', headers: { 'Content-Type': 'application/json', cookie: jar.c ?? '' }, body: JSON.stringify(body) }); + const set = res.headers.get('set-cookie'); + if (set) jar.c = set.split(';')[0]; + return { status: res.status, body: await res.json().catch(() => null) }; +} +const get = async (jar, path) => { + const res = await fetch(URL + path, { headers: { cookie: jar.c ?? '' } }); + return { status: res.status, body: await res.json().catch(() => null) }; +}; + +const cfg = await (await fetch(`${URL}/api/config`)).json(); +if (cfg.mode !== 'event') { console.error('load test needs LAB_MODE=event'); process.exit(2); } +const users = Array.from({ length: USERS }, (_, i) => ({ i, jar: {} })); + +// Sign-in provisions a workspace under the exclusive gate unless a prewarmed one is free; +// prewarm first (as an instructor would) so sign-in is a claim, not a CREATE USER. +if (process.env.PREWARM !== '0' && process.env.ADMIN_PASSWORD) { + const admin = {}; + const login = await post(admin, '/api/admin/login', { password: process.env.ADMIN_PASSWORD }); + if (login.status !== 200) { console.error('admin login failed', login); process.exit(1); } + const free = async () => (await get(admin, '/api/admin/status')).body.attendees.filter((a) => !a.email).length; + const need = USERS - await free(); + if (need > 0) { + console.log(`prewarming ${need} workspaces…`); + const r = await post(admin, '/api/admin/prewarm', { count: need }); + if (r.status !== 202) { console.error('prewarm failed', r); process.exit(1); } + const t = Date.now(); + for (let n = await free(); n < USERS; n = await free()) { + if (Date.now() - t > 90 * 60_000) { console.error(`prewarm stalled at ${n}/${USERS}`); process.exit(1); } + process.stdout.write(`\r ${n}/${USERS} ready (${Math.round((Date.now() - t) / 1000)} s)`); + await sleep(5000); + } + console.log(`\n prewarmed in ${Math.round((Date.now() - t) / 1000)} s`); + } +} + +console.log(`signing in ${USERS} attendees…`); +const signin = { ms: [], status429: 0 }; +for (const u of users) { + for (let attempt = 0; ; attempt++) { + const s = Date.now(); + const r = await post(u.jar, '/api/signin', { name: `Load ${u.i}`, email: `load${u.i}@example.com`, code: process.env.EVENT_CODE }); + if (r.status === 200) { signin.ms.push(Date.now() - s); break; } + if (r.status === 429 && attempt < 5) { signin.status429++; continue; } + console.error('sign-in failed', r); process.exit(1); + } +} +// /api/patterns needs a session in event mode, so read it as the first attendee. +const patterns = (await get(users[0].jar, '/api/patterns')).body; +const cards = patterns.find((p) => p.id === PATTERN).cards.converged; +const read = cards.find((c) => /^\s*(SELECT|WITH)\b/i.test(c.sql)).sql; + +console.log('warming (builds pattern 01 in every workspace, one at a time)…'); +for (const u of users) { + const r = await post(u.jar, '/api/run', { lane: 'sql', patternId: PATTERN, text: read }); + if (r.status !== 200) { console.error('warm-up failed', r.status, r.body); process.exit(1); } +} + +const conn = await oracledb.getConnection({ user: 'lab_admin', password: process.env.LAB_ADMIN_PASSWORD ?? 'LabAdmin2026', connectString: `${process.env.DB_HOST ?? 'localhost'}:${process.env.DB_PORT ?? 1522}/FREEPDB1` }); +let maxActive = 0; let samples = 0; let sampling = true; +const sampler = (async () => { + while (sampling) { + const r = await conn.execute("SELECT COUNT(*) FROM v$session WHERE username LIKE 'WS\\_%' ESCAPE '\\' AND status = 'ACTIVE'"); + maxActive = Math.max(maxActive, r.rows[0][0]); + samples++; + await new Promise((res) => setTimeout(res, 50)); + } +})(); + +console.log(`burst: ${USERS} attendees click Run at once (${Math.round(WRITERS * 100)}% write)…`); +const t0 = Date.now(); +const outcomes = await Promise.all(users.map(async (u) => { + const write = u.i < USERS * WRITERS; + const text = write ? `UPDATE xr_advisors SET office = 'LOAD-${u.i}' WHERE ROWNUM = 1` : read; + const s = Date.now(); + const r = await post(u.jar, '/api/run', { lane: 'sql', patternId: PATTERN, text }); + return { status: r.status, cached: r.body?.cached === true, ms: Date.now() - s, write }; +})); +const wallMs = Date.now() - t0; +sampling = false; +await sampler; +await conn.close(); + +const lat = outcomes.filter((o) => o.status === 200).map((o) => o.ms); +const report = { + users: USERS, + signin: { status429: signin.status429, p50ms: pct(signin.ms, 0.5), p95ms: pct(signin.ms, 0.95) }, + wallMs, + statuses: outcomes.reduce((a, o) => ({ ...a, [o.status]: (a[o.status] ?? 0) + 1 }), {}), + cachedReads: outcomes.filter((o) => o.cached).length, + executed: outcomes.filter((o) => o.status === 200 && !o.cached).length, + maxActiveAttendeeSessions: maxActive, + sessionSamples: samples, + p50ms: pct(lat, 0.5), + p95ms: pct(lat, 0.95), +}; +console.log(JSON.stringify(report, null, 2)); + +const problems = []; +if (maxActive > 1) problems.push(`gate breached: ${maxActive} attendee sessions active at once`); +if (Object.keys(report.statuses).some((s) => Number(s) >= 500)) problems.push('server errors during burst'); +if (report.executed > USERS * WRITERS + 1) problems.push(`cache did not absorb the reads: ${report.executed} executions`); +if (problems.length) { console.error(`LOAD FAIL:\n ${problems.join('\n ')}`); process.exit(1); } +console.log('LOAD PASS'); diff --git a/labs/converged-modeling-patterns/app/test/smoke/smoke.mjs b/labs/converged-modeling-patterns/app/test/smoke/smoke.mjs new file mode 100644 index 0000000..d6118b6 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/smoke/smoke.mjs @@ -0,0 +1,115 @@ +// app/test/smoke/smoke.mjs +import fs from 'node:fs'; +import path from 'node:path'; +import { execSync } from 'node:child_process'; +import puppeteer from 'puppeteer-core'; + +const URL = process.env.LAB_URL ?? 'http://localhost:3100'; +const OUT = path.resolve(import.meta.dirname, '../artifacts'); +fs.mkdirSync(OUT, { recursive: true }); +const chrome = process.env.CHROME_PATH + ?? ['google-chrome', 'google-chrome-stable', 'chromium', 'chromium-browser'] + .map((c) => { try { return execSync(`command -v ${c}`).toString().trim(); } catch { return null; } }).find(Boolean); +if (!chrome) { console.error('no Chrome found; set CHROME_PATH'); process.exit(2); } + +const PATTERNS = ['01-extended-reference', '02-computed', '03-bucket', '04-subset', '05-tree-hierarchy', '06-outlier']; +const fail = (m) => { console.error(`SMOKE FAIL: ${m}`); process.exit(1); }; +// The console is a fixed dock over the bottom of the page (clamp(280px, 45vh, 60vh) when +// expanded, a 40 px bar when collapsed); Puppeteer's own scroll can leave a +// target under it, so scroll the target to the top of the viewport before a real click. +const press = async (page, target) => { + const el = typeof target === 'string' ? await page.$(target) : target; + if (!el) fail(`no element for ${target}`); + await el.evaluate((e) => e.scrollIntoView({ block: 'start' })); + await el.click(); +}; +const browser = await puppeteer.launch({ executablePath: chrome, headless: true, args: ['--no-sandbox'] }); +try { + for (const scheme of ['light', 'dark']) { + const page = await browser.newPage(); + await page.setViewport({ width: 1440, height: 900 }); + await page.emulateMediaFeatures([{ name: 'prefers-color-scheme', value: scheme }]); + // Headless Chrome always reports (hover: none)/(pointer: coarse) regardless of the host's + // real input devices, and CDP's media-feature emulation (used above) does not cover hover or + // pointer — so the dock's own hover gate (there by design, to avoid sticky hover on touch) + // would never engage under this runner, no matter what page.hover() does. Patch matchMedia + // for just those two features before the app boots, so this run reflects the mouse-driven + // desktop behaviour the dock actually ships for. + await page.evaluateOnNewDocument(() => { + const native = window.matchMedia.bind(window); + window.matchMedia = (q) => (/\(\s*hover\s*:\s*hover\s*\)/.test(q) || /\(\s*pointer\s*:\s*fine\s*\)/.test(q)) + ? { matches: true, media: q, addListener() {}, removeListener() {}, addEventListener() {}, removeEventListener() {}, dispatchEvent: () => true } + : native(q); + }); + const external = []; + page.on('request', (r) => { if (!r.url().startsWith(URL) && !r.url().startsWith('data:')) external.push(r.url()); }); + await page.goto(URL, { waitUntil: 'networkidle0' }); + const cards = await page.$$('.pcard'); + if (cards.length !== 6) fail(`expected 6 pattern cards, got ${cards.length}`); + await page.screenshot({ path: path.join(OUT, `home-${scheme}.png`) }); + for (const id of PATTERNS) { + await page.goto(`${URL}/#/p/${id}`, { waitUntil: 'networkidle0' }); + await page.waitForSelector('.card'); + // Dock: collapsed on arrival, expands on hover, and on Load. + const expanded = () => page.$eval('#console', (e) => e.classList.contains('expanded')); + if (await expanded()) fail(`${id}: dock should start collapsed`); + await page.hover('#console .bar'); await new Promise((r) => setTimeout(r, 300)); + if (!(await expanded())) fail(`${id}: dock did not expand on hover`); + await page.mouse.move(10, 10); await new Promise((r) => setTimeout(r, 900)); + if (await expanded()) fail(`${id}: dock did not collapse after leaving`); + // Help: a card ⓘ and a tab ⓘ open popovers with text. + for (const sel of ['.card h3 .help-i', '#view .tabs .help-i']) { + const t = await page.$(sel); + if (!t) fail(`${id}: no help trigger for ${sel}`); + await t.click(); + const txt = await page.$eval('.help-pop', (e) => e.textContent.trim()); + if (txt.length < 20) fail(`${id}: help popover for ${sel} is empty`); + // One card's popover, left open for the screenshot, so a reviewer can see it in place. + if (sel === '.card h3 .help-i') await page.screenshot({ path: path.join(OUT, `${id}-help-${scheme}.png`) }); + await page.keyboard.press('Escape'); + } + const runButtons = await page.$$('.card .actions button:nth-child(3)'); + await press(page, runButtons[0]); + await page.waitForSelector('.card-out .result', { timeout: 60000 }); + const errs = await page.$$('.card-out .result.error'); + if (errs.length) fail(`${id}: first card returned an error`); + const loadButtons = await page.$$('.card .actions button:nth-child(2)'); + await press(page, loadButtons[0]); + await page.$eval('#out', (o) => o.replaceChildren()); // drop the previous pattern's console result + await page.click('#run'); + await page.waitForSelector('#out .result', { timeout: 60000 }); + if (await page.$('#out .result.error')) fail(`${id}: console run returned an error`); + await page.screenshot({ path: path.join(OUT, `${id}-cards-${scheme}.png`) }); + await page.setViewport({ width: 800, height: 900 }); + await page.screenshot({ path: path.join(OUT, `${id}-cards-${scheme}-800.png`) }); + await page.setViewport({ width: 1440, height: 900 }); + // Dock: Load into console (above) opened it via focus, and it should still be open. + if (!(await expanded())) fail(`${id}: dock did not stay open after Load into console`); + // Move the pointer off the dock, then click the page heading to move focus out too — + // with neither hover nor focus on it, it should close on its own after ~600ms. + await page.mouse.move(10, 10); + await press(page, '#view h1'); + await new Promise((r) => setTimeout(r, 900)); + if (await expanded()) fail(`${id}: dock did not close after focus and hover both left`); + const tabs = await page.$$('#view .tabs button.tab'); // pattern tabs, not the console's SQL/MongoDB lane tabs + await press(page, tabs[tabs.length - 1]); // Measure it + await press(page, '.card.measure .btn.primary'); + await page.waitForSelector('.measure-out svg', { timeout: 60000 }); + if (!(await page.$('.measure-out svg.curve circle.live-redo'))) fail(`${id}: measure curve has no live marker`); + if (!(await page.$('.measure-out .stat-bars'))) fail(`${id}: measure stat bars missing`); + if (await page.$('.measure-out .result.error')) fail(`${id}: measure-it returned an error`); + // fullPage renders the fixed console dock mid-page, over the measure result: hide it for this shot + await page.$eval('#console', (c) => { c.hidden = true; }); + await page.screenshot({ path: path.join(OUT, `${id}-measure-${scheme}.png`), fullPage: true }); + await page.setViewport({ width: 800, height: 900 }); + await page.screenshot({ path: path.join(OUT, `${id}-measure-${scheme}-800.png`), fullPage: true }); + await page.setViewport({ width: 1440, height: 900 }); + await page.$eval('#console', (c) => { c.hidden = false; }); + } + if (external.length) fail(`external requests: ${external.join(', ')}`); + await page.close(); + } + console.log(`SMOKE PASS · screenshots in ${OUT}`); +} finally { + await browser.close(); +} diff --git a/labs/converged-modeling-patterns/app/test/unit/auth.test.js b/labs/converged-modeling-patterns/app/test/unit/auth.test.js new file mode 100644 index 0000000..7db5ce5 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/auth.test.js @@ -0,0 +1,22 @@ +import { describe, it, expect } from 'vitest'; +import { sign, verify, parseCookies, cookie } from '../../src/services/auth.js'; + +describe('auth', () => { + it('round-trips a signed value', () => { + expect(verify(sign('WS_ABC123', 's3cret'), 's3cret')).toBe('WS_ABC123'); + }); + it('rejects tampering and wrong secrets', () => { + const t = sign('WS_ABC123', 's3cret'); + expect(verify(t.replace('ABC', 'XYZ'), 's3cret')).toBeNull(); + expect(verify(t, 'other')).toBeNull(); + expect(verify('garbage', 's3cret')).toBeNull(); + expect(verify(undefined, 's3cret')).toBeNull(); + }); + it('parses cookie headers', () => { + expect(parseCookies('a=1; lab_sid=x.y; b=%20z')).toEqual({ a: '1', lab_sid: 'x.y', b: ' z' }); + expect(parseCookies(undefined)).toEqual({}); + }); + it('builds an HttpOnly cookie', () => { + expect(cookie('lab_sid', 'v', { maxAgeSec: 60 })).toBe('lab_sid=v; Max-Age=60; Path=/; HttpOnly; SameSite=Lax'); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/cache.test.js b/labs/converged-modeling-patterns/app/test/unit/cache.test.js new file mode 100644 index 0000000..4ce139b --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/cache.test.js @@ -0,0 +1,69 @@ +import { describe, it, expect } from 'vitest'; +import { normalizeSql, isReadOnlySql, isReadOnlyMongo, patternsTouched, ResultCache } from '../../src/cache.js'; + +const ALL = ['01-extended-reference', '02-computed', '03-bucket', '04-subset', '05-tree-hierarchy', '06-outlier']; + +describe('eligibility', () => { + it('normalizes whitespace, comments and trailing semicolons but not case', () => { + expect(normalizeSql("SELECT a -- c\n FROM t WHERE x = 'Ab';")).toBe("SELECT a FROM t WHERE x = 'Ab'"); + }); + it('treats single SELECT/WITH as read-only', () => { + expect(isReadOnlySql(['SELECT 1 FROM dual'])).toBe(true); + expect(isReadOnlySql(['with q as (select 1 x from dual) select x from q'])).toBe(true); + }); + it('rejects writes, multiple statements and FOR UPDATE', () => { + expect(isReadOnlySql(['UPDATE t SET a = 1'])).toBe(false); + expect(isReadOnlySql(['SELECT 1 FROM dual', 'SELECT 2 FROM dual'])).toBe(false); + expect(isReadOnlySql(['SELECT * FROM t FOR UPDATE'])).toBe(false); + expect(isReadOnlySql(['BEGIN NULL; END;'])).toBe(false); + }); + it('classifies Mongo commands', () => { + expect(isReadOnlyMongo({ kind: 'showCollections' })).toBe(true); + expect(isReadOnlyMongo({ kind: 'collection', op: 'find', args: [{}], mods: {} })).toBe(true); + expect(isReadOnlyMongo({ kind: 'collection', op: 'updateOne', args: [{}, {}], mods: {} })).toBe(false); + expect(isReadOnlyMongo({ kind: 'collection', op: 'aggregate', args: [[{ $match: {} }, { $out: 'x' }]], mods: {} })).toBe(false); + expect(isReadOnlyMongo({ kind: 'db', op: 'aggregate', args: [[{ $sql: 'select 1 from dual' }]], mods: {} })).toBe(true); + expect(isReadOnlyMongo({ kind: 'db', op: 'aggregate', args: [[{ $sql: 'delete from t' }]], mods: {} })).toBe(false); + }); + it('classifies the object form of $sql by its statement; a missing statement is a write', () => { + const agg = (v) => isReadOnlyMongo({ kind: 'db', op: 'aggregate', args: [[{ $sql: v }]], mods: {} }); + expect(agg({ statement: 'select 1 from dual' })).toBe(true); + expect(agg({ statement: 'delete from t' })).toBe(false); + expect(agg({ statement: 'select 1 from dual for update' })).toBe(false); + expect(agg({})).toBe(false); + expect(agg({ statement: 42 })).toBe(false); + expect(agg(null)).toBe(false); + }); + it('maps statements to the patterns whose objects they touch', () => { + expect(patternsTouched('UPDATE xr_advisors SET office = 1', ALL)).toEqual(['01-extended-reference']); + expect(patternsTouched('insert into bk_readings select * from sb_claim_events', ALL).sort()).toEqual(['03-bucket', '04-subset']); + expect(patternsTouched('create table scratch (a number)', ALL)).toEqual(ALL); + }); +}); + +describe('ResultCache', () => { + it('keys on pattern, trimmed raw text and dataset version', () => { + const c = new ResultCache(); + expect(c.key('p', ' SELECT 1 FROM dual \n', 'v1')).toBe(c.key('p', 'SELECT 1 FROM dual', 'v1')); + expect(c.key('p', 'SELECT 1 FROM dual', 'v1')).not.toBe(c.key('p', 'SELECT 1 FROM dual', 'v1')); + expect(c.key('p', 'SELECT 1 FROM dual;', 'v1')).not.toBe(c.key('p', 'SELECT 1 FROM dual', 'v1')); + expect(c.key('p', '--x', 'v1')).not.toBe(c.key('p', '--y', 'v1')); + expect(c.key('p', 'SELECT 1 FROM dual', 'v1')).not.toBe(c.key('p', 'SELECT 1 FROM dual', 'v2')); + expect(c.key('p', 'SELECT 1 FROM dual', 'v1')).not.toBe(c.key('q', 'SELECT 1 FROM dual', 'v1')); + }); + it('evicts the least recently used entry', () => { + const c = new ResultCache({ maxEntries: 2 }); + c.set('a', 1); c.set('b', 2); + c.get('a'); + c.set('c', 3); + expect(c.get('b')).toBeUndefined(); + expect(c.get('a')).toBe(1); + expect(c.size).toBe(2); + }); + it('returns nothing while disabled', () => { + const c = new ResultCache(); + c.set('a', 1); + c.enabled = false; + expect(c.get('a')).toBeUndefined(); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/chart.test.js b/labs/converged-modeling-patterns/app/test/unit/chart.test.js new file mode 100644 index 0000000..fe9bfdf --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/chart.test.js @@ -0,0 +1,68 @@ +// @vitest-environment jsdom +import { describe, it, expect } from 'vitest'; +import { curveChart, logScale } from '../../public/js/chart.js'; + +const CAL = [{ x: 10, ratio: 1.2 }, { x: 100, ratio: 5 }, { x: 1000, ratio: 33 }, { x: 5000, ratio: 260 }]; + +describe('logScale', () => { + it('maps decades linearly', () => { + const s = logScale([1, 1000], [0, 300]); + expect(s(1)).toBeCloseTo(0); expect(s(10)).toBeCloseTo(100); expect(s(1000)).toBeCloseTo(300); + }); + it('ticks are powers of ten inside the domain', () => { + expect(logScale([3, 5000], [0, 1]).ticks()).toEqual([10, 100, 1000]); + }); +}); + +describe('curveChart', () => { + it('draws the calibration line, points, break-even line and a live marker', () => { + const svg = curveChart({ calibration: CAL, labX: 1000, xLabel: 'CDR items', live: { redo: 32.8, blocks: 1.9 } }); + expect(svg.querySelector('path.cal')).not.toBeNull(); + expect(svg.querySelectorAll('circle.calpt').length).toBe(4); + expect(svg.querySelector('line.breakeven')).not.toBeNull(); + expect(svg.querySelector('circle.live-redo')).not.toBeNull(); + expect(svg.querySelector('circle.live-blocks')).not.toBeNull(); + expect(svg.textContent).toContain('you · 32.8× redo'); + }); + it('no marker when the live ratio is not finite', () => { + const svg = curveChart({ calibration: CAL, labX: 1000, xLabel: 'x', live: { redo: Infinity, blocks: null } }); + expect(svg.querySelector('circle.live-redo')).toBeNull(); + expect(svg.querySelector('circle.live-blocks')).toBeNull(); + expect(svg.textContent).not.toMatch(/NaN|Infinity/); + }); + it('offsets the live markers apart when they would otherwise coincide with a calibration point', () => { + // labX is the first calibration x, and both live ratios equal that point's reference ratio — + // both the "near-equal to each other" and "on top of a calibration point" triggers fire. + const svg = curveChart({ calibration: CAL, labX: 10, xLabel: 'x', live: { redo: 1.2, blocks: 1.2 } }); + const dot = svg.querySelector('circle.live-redo'); + const ring = svg.querySelector('circle.live-blocks'); + expect(dot.getAttribute('cx')).not.toBe(ring.getAttribute('cx')); + }); + it('anchors the live label at the end when labX sits at the maximum calibration x', () => { + // Live ratio kept off the 5000 calibration point: that point is now a label obstacle, and a + // label sitting on it would (correctly) move above instead of anchoring at the end. + const svg = curveChart({ calibration: CAL, labX: 5000, xLabel: 'x', live: { redo: 120, blocks: null } }); + const label = [...svg.querySelectorAll('text')].find((el) => el.textContent.startsWith('you')); + expect(label.getAttribute('text-anchor')).toBe('end'); + expect(Number(label.getAttribute('x'))).toBeLessThanOrEqual(540); + }); + + it('moves the break-even label off a calibration point that sits under its default position', () => { + // 02-computed's real calibration: the x=10 point (1.73x) sits just above the break-even line, + // right where the left-anchored label used to go. Box estimate matches chart.js (0.6 em/char). + const cal = [{ x: 10, ratio: 1.73 }, { x: 100, ratio: 7.16 }, { x: 1000, ratio: 32.53 }, { x: 5000, ratio: 203.24 }]; + const svg = curveChart({ calibration: cal, labX: 1000, xLabel: 'x', live: { redo: 32.8, blocks: 2 } }); + const be = [...svg.querySelectorAll('text')].find((el) => el.textContent.startsWith('break-even')); + const fs = Number(be.getAttribute('font-size')); const w = 0.6 * fs * be.textContent.length; + const bx = Number(be.getAttribute('x')); const by = Number(be.getAttribute('y')); + const anchor = be.getAttribute('text-anchor'); + const x0 = anchor === 'end' ? bx - w : anchor === 'middle' ? bx - w / 2 : bx; + const box = { x0, x1: x0 + w, y0: by - fs, y1: by + fs * 0.3 }; + const pt = svg.querySelector('circle.calpt'); // first = x=10 + const cx = Number(pt.getAttribute('cx')); const cy = Number(pt.getAttribute('cy')); const r = Number(pt.getAttribute('r')); + // Distance from the point's edge to the label box. + const dx = Math.max(box.x0 - cx, 0, cx - box.x1); const dy = Math.max(box.y0 - cy, 0, cy - box.y1); + expect(Math.hypot(dx, dy) - r).toBeGreaterThanOrEqual(8); + }); +}); + diff --git a/labs/converged-modeling-patterns/app/test/unit/config.test.js b/labs/converged-modeling-patterns/app/test/unit/config.test.js new file mode 100644 index 0000000..3d63690 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/config.test.js @@ -0,0 +1,65 @@ +import { describe, it, expect } from 'vitest'; +import { loadConfig } from '../../src/config.js'; + +describe('loadConfig', () => { + it('defaults to solo mode with a single-permit gate and pools of 1', () => { + const c = loadConfig({}); + expect(c.mode).toBe('solo'); + expect(c.port).toBe(3000); + expect(c.gate).toEqual({ permits: 1, queueTimeoutMs: 30000, sqlTimeoutMs: 10000, mongoTimeoutMs: 10000 }); + expect(c.db.poolMax).toBe(1); + expect(c.mongo.poolMax).toBe(1); + expect(c.cache).toEqual({ maxEntries: 500, enabled: true }); + expect(c.limits).toEqual({ maxRows: 500, maxBytes: 1048576, maxStatements: 20 }); + expect(c.db.host).toBe('oracle'); + expect(c.db.service).toBe('FREEPDB1'); + }); + + it('reads overrides from the environment', () => { + const c = loadConfig({ DB_POOL_MAX: '2', MONGO_POOL_MAX: '3', DB_HOST: 'x', SQL_TIMEOUT_MS: '5000' }); + expect(c.db.poolMax).toBe(2); + expect(c.mongo.poolMax).toBe(3); + expect(c.db.host).toBe('x'); + expect(c.gate.sqlTimeoutMs).toBe(5000); + }); + + it('rejects an unknown mode', () => { + expect(() => loadConfig({ LAB_MODE: 'party' })).toThrow(/LAB_MODE/); + }); + + it('requires ADMIN_PASSWORD in event mode', () => { + expect(() => loadConfig({ LAB_MODE: 'event', LAB_ADMIN_PASSWORD: 'Own-LabAdmin-1' })).toThrow(/ADMIN_PASSWORD/); + const c = loadConfig({ LAB_MODE: 'event', ADMIN_PASSWORD: 'pw', EVENT_CODE: 'NW26', LAB_ADMIN_PASSWORD: 'Own-LabAdmin-1' }); + expect(c.event).toEqual({ code: 'NW26', adminPassword: 'pw' }); + }); + + it('refuses to start event mode while LAB_ADMIN_PASSWORD is the repo default', () => { + expect(() => loadConfig({ LAB_MODE: 'event', ADMIN_PASSWORD: 'pw' })).toThrow(/LAB_ADMIN_PASSWORD.*default/); + expect(() => loadConfig({ LAB_MODE: 'event', ADMIN_PASSWORD: 'pw', LAB_ADMIN_PASSWORD: 'LabAdmin2026' })).toThrow(/LAB_ADMIN_PASSWORD/); + expect(loadConfig({ LAB_MODE: 'event', ADMIN_PASSWORD: 'pw', LAB_ADMIN_PASSWORD: 'Own-LabAdmin-1' }).db.labAdminPassword).toBe('Own-LabAdmin-1'); + }); + + it('keeps solo mode default-friendly', () => { + expect(loadConfig({}).db.labAdminPassword).toBe('LabAdmin2026'); + expect(loadConfig({ LAB_MODE: 'solo' }).mode).toBe('solo'); + }); + + it('lets the integration harness opt out of the default-secret check', () => { + const c = loadConfig({ LAB_MODE: 'event', ADMIN_PASSWORD: 'pw' }, { allowDefaultSecrets: true }); + expect(c.db.labAdminPassword).toBe('LabAdmin2026'); + }); + + it('rejects database passwords containing a double quote', () => { + expect(() => loadConfig({ LAB_ADMIN_PASSWORD: 'a"b' })).toThrow(/LAB_ADMIN_PASSWORD must not contain "/); + expect(() => loadConfig({ ORACLE_PASSWORD: 'a"b' })).toThrow(/ORACLE_PASSWORD/); + expect(() => loadConfig({ CMP_PASSWORD: 'a"b' })).toThrow(/CMP_PASSWORD/); + }); + + it('rejects non-positive pool sizes', () => { + expect(() => loadConfig({ DB_POOL_MAX: '0' })).toThrow(/DB_POOL_MAX/); + }); + + it('is frozen', () => { + expect(Object.isFrozen(loadConfig({}))).toBe(true); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/dock.test.js b/labs/converged-modeling-patterns/app/test/unit/dock.test.js new file mode 100644 index 0000000..6226536 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/dock.test.js @@ -0,0 +1,144 @@ +// @vitest-environment jsdom +import { describe, it, expect, beforeEach, vi } from 'vitest'; +import { Dock } from '../../public/js/dock.js'; + +function mk() { + document.body.innerHTML = ''; + const root = document.createElement('section'); root.id = 'console'; + const bar = document.createElement('div'); bar.className = 'bar'; + const t = document.createElement('button'); t.id = 'dock-toggle'; + const pin = document.createElement('button'); pin.id = 'dock-pin'; + const ed = document.createElement('div'); ed.className = 'editor'; ed.tabIndex = 0; + bar.append(t, pin); root.append(bar, ed); document.body.append(root); + const mem = {}; const storage = { getItem: (k) => mem[k] ?? null, setItem: (k, v) => { mem[k] = v; } }; + return { root, t, pin, ed, storage, dock: new Dock(root, { storage }) }; +} + +describe('Dock', () => { + beforeEach(() => vi.useFakeTimers()); + + it('starts collapsed with aria-expanded=false', () => { + const { root, t } = mk(); + expect(root.classList.contains('expanded')).toBe(false); + expect(t.getAttribute('aria-expanded')).toBe('false'); + }); + it('opens on hover after the intent delay and closes 600 ms after leaving', () => { + const { root } = mk(); + root.dispatchEvent(new Event('pointerenter')); vi.advanceTimersByTime(149); + expect(root.classList.contains('expanded')).toBe(false); + vi.advanceTimersByTime(1); + expect(root.classList.contains('expanded')).toBe(true); + root.dispatchEvent(new Event('pointerleave')); vi.advanceTimersByTime(599); + expect(root.classList.contains('expanded')).toBe(true); + vi.advanceTimersByTime(1); + expect(root.classList.contains('expanded')).toBe(false); + }); + it('stays open while focused, busy or pinned', () => { + const { root, ed, dock } = mk(); + dock.open('hover'); + ed.focus(); root.dispatchEvent(new Event('focusin')); + root.dispatchEvent(new Event('pointerleave')); vi.advanceTimersByTime(2000); + expect(root.classList.contains('expanded')).toBe(true); + ed.blur(); root.dispatchEvent(new Event('focusout')); + dock.setBusy(true); root.dispatchEvent(new Event('pointerleave')); vi.advanceTimersByTime(2000); + expect(root.classList.contains('expanded')).toBe(true); + dock.setBusy(false); dock.togglePin(); root.dispatchEvent(new Event('pointerleave')); vi.advanceTimersByTime(2000); + expect(root.classList.contains('expanded')).toBe(true); + }); + it('remembers the pin in storage and tolerates storage that throws', () => { + const { storage, dock } = mk(); + dock.togglePin(); + expect(storage.getItem('lab.dock.pinned')).toBe('true'); + const bad = { getItem: () => { throw new Error('x'); }, setItem: () => { throw new Error('x'); } }; + expect(() => new Dock(document.getElementById('console'), { storage: bad }).togglePin()).not.toThrow(); + }); + it('the toggle button opens and closes explicitly', () => { + const { root, t } = mk(); + t.click(); expect(root.classList.contains('expanded')).toBe(true); + t.click(); expect(root.classList.contains('expanded')).toBe(false); + }); + + it('closes 600ms after focus leaves to outside the dock, with the pointer not on it (Load into console, then click away)', () => { + const { root, ed, dock } = mk(); + dock.open('load'); + ed.focus(); root.dispatchEvent(new Event('focusin')); + ed.blur(); root.dispatchEvent(new Event('focusout')); // relatedTarget is outside root + vi.advanceTimersByTime(599); + expect(root.classList.contains('expanded')).toBe(true); + vi.advanceTimersByTime(1); + expect(root.classList.contains('expanded')).toBe(false); + }); + + it('stays open if focus leaves the dock while the pointer is still hovering it', () => { + const { root, ed } = mk(); + root.dispatchEvent(new Event('pointerenter')); vi.advanceTimersByTime(150); + expect(root.classList.contains('expanded')).toBe(true); + ed.focus(); root.dispatchEvent(new Event('focusin')); + ed.blur(); root.dispatchEvent(new Event('focusout')); + vi.advanceTimersByTime(2000); + expect(root.classList.contains('expanded')).toBe(true); + }); + + it('closes 600ms after a run finishes (setBusy(false)) with no hover and no focus', () => { + const { root, dock } = mk(); + dock.setBusy(true); + expect(root.classList.contains('expanded')).toBe(true); + dock.setBusy(false); + vi.advanceTimersByTime(599); + expect(root.classList.contains('expanded')).toBe(true); + vi.advanceTimersByTime(1); + expect(root.classList.contains('expanded')).toBe(false); + }); + + it('never closes while pinned, even after a run finishes with no hover or focus', () => { + const { root, dock } = mk(); + dock.togglePin(); + dock.setBusy(true); + dock.setBusy(false); + vi.advanceTimersByTime(2000); + expect(root.classList.contains('expanded')).toBe(true); + }); + + it('Esc from the editor collapses the dock and leaves focus on the toggle without reopening it', () => { + const { root, t, ed, dock } = mk(); + dock.open('load'); + ed.focus(); ed.dispatchEvent(new FocusEvent('focusin', { bubbles: true })); + ed.dispatchEvent(new KeyboardEvent('keydown', { key: 'Escape', bubbles: true })); + t.dispatchEvent(new FocusEvent('focusin', { bubbles: true })); + vi.advanceTimersByTime(2000); + expect(document.activeElement).toBe(t); + expect(root.classList.contains('expanded')).toBe(false); + }); + + it('on a touch device (hover: none), focus then click on the toggle toggles exactly once per click', () => { + const orig = globalThis.matchMedia; + globalThis.matchMedia = () => ({ matches: false }); + try { + const { root, t, dock } = mk(); + expect(dock.hoverable).toBe(false); + t.focus(); t.dispatchEvent(new FocusEvent('focusin', { bubbles: true })); t.click(); + expect(root.classList.contains('expanded')).toBe(true); + t.click(); + expect(root.classList.contains('expanded')).toBe(false); + } finally { globalThis.matchMedia = orig; } + }); + + it('keyboard focus on the toggle or pin alone does not expand the dock; focus elsewhere inside does', () => { + const { root, t, pin, ed } = mk(); + t.focus(); t.dispatchEvent(new FocusEvent('focusin', { bubbles: true })); + expect(root.classList.contains('expanded')).toBe(false); + pin.focus(); pin.dispatchEvent(new FocusEvent('focusin', { bubbles: true })); + expect(root.classList.contains('expanded')).toBe(false); + ed.focus(); ed.dispatchEvent(new FocusEvent('focusin', { bubbles: true })); + expect(root.classList.contains('expanded')).toBe(true); + }); + + it('pinning toggles the dock-pinned class on body so the page can pad for the dock', () => { + const { dock } = mk(); + expect(document.body.classList.contains('dock-pinned')).toBe(false); + dock.togglePin(); + expect(document.body.classList.contains('dock-pinned')).toBe(true); + dock.togglePin(); + expect(document.body.classList.contains('dock-pinned')).toBe(false); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/execPool.test.js b/labs/converged-modeling-patterns/app/test/unit/execPool.test.js new file mode 100644 index 0000000..6561e3a --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/execPool.test.js @@ -0,0 +1,94 @@ +import { describe, it, expect, vi, afterEach } from 'vitest'; +import { createExecPool } from '../../src/db/oracle.js'; + +const tick = () => new Promise((r) => setImmediate(r)); + +function makeConn() { + let closeCount = 0; + return { + closed: false, + lastCloseArgs: undefined, + async close(...args) { + closeCount++; + this.lastCloseArgs = args[0]; + if (closeCount > 1) throw new Error('NJS-003: invalid connection'); // mirrors real driver's ERR_INVALID_CONNECTION + this.closed = true; + }, + }; +} + +const cfg = (poolMax = 1, queueTimeoutMs = 1000) => ({ db: { poolMax }, gate: { queueTimeoutMs } }); + +afterEach(() => vi.useRealTimers()); + +describe('createExecPool', () => { + it('caps concurrent connections at poolMax — a second getConnection waits', async () => { + const pool = createExecPool(cfg(1), { connect: async () => makeConn() }); + const c1 = await pool.getConnection(); + let resolved = false; + const p2 = pool.getConnection().then((c) => { resolved = true; return c; }); + await tick(); + expect(resolved).toBe(false); + await c1.close(); + const c2 = await p2; + expect(resolved).toBe(true); + expect(c2).toBeDefined(); + }); + + it('rejects a queued getConnection at queueTimeoutMs', async () => { + vi.useFakeTimers(); + const pool = createExecPool(cfg(1, 500), { connect: async () => makeConn() }); + await pool.getConnection(); // holds the only slot + const waiting = pool.getConnection(); + const assertion = expect(waiting).rejects.toThrow(/timed out/); + await vi.advanceTimersByTimeAsync(500); + await assertion; + }); + + it('returns the slot after close({ drop: true })', async () => { + const pool = createExecPool(cfg(1), { connect: async () => makeConn() }); + const c1 = await pool.getConnection(); + await c1.close({ drop: true }); + expect(c1.lastCloseArgs).toEqual({ drop: true }); + let resolved = false; + pool.getConnection().then(() => { resolved = true; }); + await tick(); + expect(resolved).toBe(true); // slot was free, no wait needed + }); + + it('a double close releases only once — the cap still holds afterwards', async () => { + const pool = createExecPool(cfg(1), { connect: async () => makeConn() }); + const c1 = await pool.getConnection(); + await c1.close(); + await c1.close().catch(() => {}); // real driver rejects the second close; callers often swallow it + const c2 = await pool.getConnection(); // one slot was freed by the (single) release — this succeeds + let resolved = false; + pool.getConnection().then(() => { resolved = true; }); + await tick(); + expect(resolved).toBe(false); // a second release would have wrongly freed a slot for this one too + expect(c2).toBeDefined(); + }); + + it('a failed connect returns the slot', async () => { + const connect = vi.fn() + .mockRejectedValueOnce(new Error('ORA-01017: invalid username/password; logon denied')) + .mockResolvedValue(makeConn()); + const pool = createExecPool(cfg(1), { connect }); + await expect(pool.getConnection()).rejects.toThrow(/ORA-01017/); + let resolved = false; + pool.getConnection().then(() => { resolved = true; }); + await tick(); + expect(resolved).toBe(true); // the failed attempt did not leak the slot + }); + + it('close() rejects a queued waiter and later getConnection() calls', async () => { + const pool = createExecPool(cfg(1), { connect: async () => makeConn() }); + const c1 = await pool.getConnection(); // holds the only slot + const waiting = pool.getConnection(); + const waitingAssertion = expect(waiting).rejects.toThrow(/closing/); + await pool.close(); + await waitingAssertion; + expect(c1.closed).toBe(true); // tracked live connections are closed on shutdown + await expect(pool.getConnection()).rejects.toThrow(/closed/); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/figures.test.js b/labs/converged-modeling-patterns/app/test/unit/figures.test.js new file mode 100644 index 0000000..e2930c5 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/figures.test.js @@ -0,0 +1,37 @@ +import { describe, it, expect } from 'vitest'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import express from 'express'; +import request from 'supertest'; +import { figuresRouter } from '../../src/routes/figures.js'; + +function app() { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'fig-')); + fs.mkdirSync(path.join(root, '02-computed', 'figures'), { recursive: true }); + fs.writeFileSync(path.join(root, '02-computed', 'figures', 'erd.svg'), ''); + fs.writeFileSync(path.join(root, 'secret.svg'), ''); + const a = express(); + a.use('/figures', figuresRouter({ cfg: { patternsDir: root }, patterns: [{ id: '02-computed' }] })); + return a; +} + +describe('GET /figures/:pattern/:file', () => { + it('serves a known figure as svg', async () => { + const r = await request(app()).get('/figures/02-computed/erd.svg'); + expect(r.status).toBe(200); + expect(r.headers['content-type']).toMatch(/image\/svg\+xml/); + expect(r.headers['x-content-type-options']).toBe('nosniff'); + // Opened full size in its own tab, an SVG is a document: no scripts, no external loads. + expect(r.headers['content-security-policy']).toBe("default-src 'none'; style-src 'unsafe-inline'"); + }); + it.each([ + '/figures/99-nope/erd.svg', + '/figures/02-computed/missing.svg', + '/figures/02-computed/..%2Fsecret.svg', + '/figures/02-computed/ERD.SVG', + '/figures/02-computed/erd.svg.js', + ])('404 for %s', async (u) => { + expect((await request(app()).get(u)).status).toBe(404); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/gate.test.js b/labs/converged-modeling-patterns/app/test/unit/gate.test.js new file mode 100644 index 0000000..b02ac5d --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/gate.test.js @@ -0,0 +1,125 @@ +import { describe, it, expect, vi, afterEach } from 'vitest'; +import { Gate, GateError } from '../../src/gate.js'; + +const deferred = () => { let resolve, reject; const p = new Promise((a, b) => { resolve = a; reject = b; }); return { p, resolve, reject }; }; +const tick = () => new Promise((r) => setImmediate(r)); + +afterEach(() => vi.useRealTimers()); + +describe('Gate', () => { + it('runs one operation at a time with one permit', async () => { + const g = new Gate({ permits: 1 }); + let active = 0; let max = 0; + const d = [deferred(), deferred(), deferred()]; + const ps = d.map((x, i) => g.run({ userId: `u${i}`, label: 'q' }, async () => { active++; max = Math.max(max, active); await x.p; active--; return i; })); + await tick(); + expect(active).toBe(1); + d.forEach((x) => x.resolve()); + expect(await Promise.all(ps)).toEqual([0, 1, 2]); + expect(max).toBe(1); + }); + + it('is first-in first-out across users', async () => { + const g = new Gate({ permits: 1 }); + const order = []; + const hold = deferred(); + const first = g.run({ userId: 'a', label: 'q' }, async () => { await hold.p; order.push('a'); }); + const rest = ['b', 'c', 'd'].map((u) => g.run({ userId: u, label: 'q' }, async () => { order.push(u); })); + hold.resolve(); + await Promise.all([first, ...rest]); + expect(order).toEqual(['a', 'b', 'c', 'd']); + }); + + it('makes an exclusive request wait for every permit and blocks later requests', async () => { + const g = new Gate({ permits: 2 }); + const order = []; const holdA = deferred(); + const a = g.run({ userId: 'a', label: 'n' }, async () => { order.push('a+'); await holdA.p; order.push('a-'); }); + const x = g.run({ userId: 'x', label: 'm', exclusive: true }, async () => { order.push('x'); }); + const b = g.run({ userId: 'b', label: 'n' }, async () => { order.push('b'); }); + await tick(); + expect(order).toEqual(['a+']); + holdA.resolve(); + await Promise.all([a, x, b]); + expect(order).toEqual(['a+', 'a-', 'x', 'b']); + }); + + it('allows only one queued request per user', async () => { + const g = new Gate({ permits: 1 }); + const hold = deferred(); + const running = g.run({ userId: 'u1', label: 'q' }, () => hold.p); + const queued = g.run({ userId: 'u2', label: 'q' }, async () => 'ok'); + await expect(g.run({ userId: 'u2', label: 'q' }, async () => 'dup')).rejects.toMatchObject({ code: 'already_queued', status: 409 }); + const u1Again = g.run({ userId: 'u1', label: 'q' }, async () => 'u1-second'); // u1 is running, not queued + hold.resolve(); + expect(await queued).toBe('ok'); + expect(await u1Again).toBe('u1-second'); + await running; + }); + + it('rejects with busy after the queue timeout', async () => { + vi.useFakeTimers(); + const g = new Gate({ permits: 1, queueTimeoutMs: 30000 }); + g.run({ userId: 'a', label: 'q' }, () => new Promise(() => {})); + const waiting = g.run({ userId: 'b', label: 'q' }, async () => 'never'); + vi.advanceTimersByTime(30000); + await expect(waiting).rejects.toMatchObject({ code: 'busy', status: 429 }); + expect(g.status().queued).toHaveLength(0); + }); + + it('pause rejects queued and new requests; resume accepts again', async () => { + const g = new Gate({ permits: 1 }); + const hold = deferred(); + const running = g.run({ userId: 'a', label: 'q' }, () => hold.p); + const queued = g.run({ userId: 'b', label: 'q' }, async () => 1); + g.pause(true); + await expect(queued).rejects.toMatchObject({ code: 'paused', status: 423 }); + await expect(g.run({ userId: 'c', label: 'q' }, async () => 1)).rejects.toBeInstanceOf(GateError); + g.pause(false); + hold.resolve(); await running; + expect(await g.run({ userId: 'c', label: 'q' }, async () => 2)).toBe(2); + }); + + it('reports queue position', async () => { + const g = new Gate({ permits: 1 }); + const hold = deferred(); + g.run({ userId: 'a', label: 'q' }, () => hold.p); + g.run({ userId: 'b', label: 'q' }, async () => 1); + g.run({ userId: 'c', label: 'q' }, async () => 1); + await tick(); + expect(g.position('b')).toBe(0); + expect(g.position('c')).toBe(1); + expect(g.position('a')).toBe(null); + hold.resolve(); + }); + + it('releases the permit when the operation throws', async () => { + const g = new Gate({ permits: 1 }); + await expect(g.run({ userId: 'a', label: 'q' }, async () => { throw new Error('ORA-00942'); })).rejects.toThrow('ORA-00942'); + expect(await g.run({ userId: 'b', label: 'q' }, async () => 'next')).toBe('next'); + }); + + it('cancel calls the running operation cancel hook', async () => { + const g = new Gate({ permits: 1 }); + const cancel = vi.fn(); + const hold = deferred(); + const p = g.run({ userId: 'a', label: 'q' }, (ctx) => { ctx.cancel = () => { cancel(); hold.reject(new Error('cancelled')); }; return hold.p; }); + await tick(); + const [{ id }] = g.status().running; + expect(g.cancel(id)).toBe(true); + await expect(p).rejects.toThrow('cancelled'); + expect(cancel).toHaveBeenCalledOnce(); + }); + + it('cancel sets ctx.cancelled for statement loops to check', async () => { + const g = new Gate({ permits: 1 }); + const hold = deferred(); + let seen; + const p = g.run({ userId: 'a', label: 'q' }, async (ctx) => { ctx.cancel = () => {}; await hold.p; seen = ctx.cancelled; }); + await tick(); + const [{ id }] = g.status().running; + expect(g.cancel(id)).toBe(true); + hold.resolve(); + await p; + expect(seen).toBe(true); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/guard.test.js b/labs/converged-modeling-patterns/app/test/unit/guard.test.js new file mode 100644 index 0000000..52b2b9b --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/guard.test.js @@ -0,0 +1,64 @@ +import { describe, it, expect } from 'vitest'; +import fs from 'node:fs'; +import path from 'node:path'; +import { classifySql, classifyMongo } from '../../src/guard.js'; +import { parseSqlFile } from '../../src/content/sqlParser.js'; + +describe('classifySql', () => { + it.each([ + 'SELECT * FROM xr_client_dv', + "INSERT INTO t VALUES ('{\"a\":1}')", + 'BEGIN NULL; END;', + 'CREATE OR REPLACE TRIGGER trg AFTER INSERT ON t FOR EACH ROW BEGIN NULL; END;', + "CREATE OR REPLACE JSON RELATIONAL DUALITY VIEW v AS SELECT JSON {'_id': t.id} FROM t", + 'SELECT table_name FROM user_tables', + ])('allows %s', (sql) => expect(classifySql(sql)).toEqual({ allowed: true })); + + it.each([ + ['alter system flush shared_pool', /ALTER SYSTEM/], + ['ALTER DATABASE OPEN', /ALTER DATABASE/], + ['drop user ws_abc cascade', /USER/], + ['CREATE /* x */ USER bob IDENTIFIED BY p', /USER/], + ["BEGIN EXECUTE IMMEDIATE 'GRANT DBA TO WS_1'; END;", /GRANT/], + ['REVOKE SELECT ON t FROM x', /REVOKE/], + ['create database link l connect to a identified by b using \'x\'', /DATABASE LINK/], + ["BEGIN dbms_scheduler.create_job('j'); END;", /DBMS_SCHEDULER/], + ["BEGIN DBMS_JOB.SUBMIT(:j, 'x'); END;", /DBMS_JOB/], + ["SELECT utl_http.request('http://x') FROM dual", /UTL_/], + ["BEGIN DBMS_PIPE.PURGE('p'); END;", /DBMS_PIPE/], + ["CREATE DIRECTORY d AS '/tmp'", /DIRECTORY/], + ['', /empty/i], + ])('blocks %s', (sql, reason) => { + const r = classifySql(sql); + expect(r.allowed).toBe(false); + expect(r.reason).toMatch(reason); + }); + + it('allows every statement in the pattern files', () => { + const root = path.resolve(import.meta.dirname, '../../../patterns'); + for (const dir of fs.readdirSync(root)) { + for (const f of ['01-document-model.sql', '02-converged.sql']) { + const p = path.join(root, dir, f); + if (!fs.existsSync(p)) continue; + for (const st of parseSqlFile(fs.readFileSync(p, 'utf8'))) expect(classifySql(st.sql), `${p}:${st.line}`).toEqual({ allowed: true }); + } + } + }); +}); + +describe('classifyMongo', () => { + it('checks $sql stages with the SQL rules', () => { + expect(classifyMongo({ kind: 'db', op: 'aggregate', args: [[{ $sql: 'select 1 from dual' }]], mods: {} })).toEqual({ allowed: true }); + const r = classifyMongo({ kind: 'db', op: 'aggregate', args: [[{ $sql: "begin execute immediate 'grant dba to x'; end;" }]], mods: {} }); + expect(r.allowed).toBe(false); + }); + it('checks the object form of $sql by its statement string', () => { + const agg = (v) => classifyMongo({ kind: 'db', op: 'aggregate', args: [[{ $sql: v }]], mods: {} }); + expect(agg({ statement: 'select 1 from dual' })).toEqual({ allowed: true }); + expect(agg({ statement: 'GRANT DBA TO x' })).toEqual({ allowed: false, reason: 'Blocked in this lab: GRANT' }); + expect(agg({ statement: 'alter system flush shared_pool', binds: [] }).allowed).toBe(false); + }); + it('allows ordinary collection commands', () => { + expect(classifyMongo({ kind: 'collection', collection: 'c', op: 'find', args: [{}], mods: {} })).toEqual({ allowed: true }); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/help.test.js b/labs/converged-modeling-patterns/app/test/unit/help.test.js new file mode 100644 index 0000000..8334ef5 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/help.test.js @@ -0,0 +1,139 @@ +// @vitest-environment jsdom +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { helpTrigger, placePopover } from '../../public/js/help.js'; + +describe('placePopover', () => { + const vp = { width: 1000, height: 800 }; + it('prefers below the trigger', () => { + expect(placePopover({ top: 100, bottom: 120, left: 200, right: 216 }, { width: 400, height: 200 }, vp)).toEqual({ top: 128, left: 200, side: 'below' }); + }); + it('flips above when there is no room below', () => { + expect(placePopover({ top: 700, bottom: 720, left: 200, right: 216 }, { width: 400, height: 200 }, vp).side).toBe('above'); + }); + it('clamps to a 16 px gutter on the right', () => { + expect(placePopover({ top: 100, bottom: 120, left: 900, right: 916 }, { width: 400, height: 200 }, vp).left).toBe(584); + }); +}); + +describe('helpTrigger', () => { + beforeEach(() => { document.body.innerHTML = ''; }); + it('returns null when there is nothing to say', () => { + expect(helpTrigger({ why: null, look: null, figure: null }, { label: 'x' })).toBeNull(); + }); + it('opens a dialog with text sections on click and closes on Escape', () => { + const t = helpTrigger({ why: 'Because physics.', look: 'Row count 1.', figure: { file: 'erd.svg', caption: 'The ERD' } }, { label: 'Read it', patternId: '02-computed' }); + document.body.append(t); + t.click(); + const d = document.querySelector('.help-pop'); + expect(d.getAttribute('role')).toBe('dialog'); + expect(d.textContent).toContain('Because physics.'); + expect(d.querySelector('b')).toBeNull(); + expect(d.querySelector('img').getAttribute('src')).toBe('/figures/02-computed/erd.svg'); + d.dispatchEvent(new KeyboardEvent('keydown', { key: 'Escape', bubbles: true })); + expect(document.querySelector('.help-pop')).toBeNull(); + }); + it('only one popover is open at a time', () => { + const a = helpTrigger({ why: 'a' }, { label: 'a' }); const b = helpTrigger({ why: 'b' }, { label: 'b' }); + document.body.append(a, b); a.click(); b.click(); + expect(document.querySelectorAll('.help-pop').length).toBe(1); + expect(document.querySelector('.help-pop').textContent).toContain('b'); + }); +}); + +describe('helpTrigger items', () => { + beforeEach(() => { document.body.innerHTML = ''; }); + it('returns null when items is empty and nothing else is set', () => { + expect(helpTrigger({ items: [] }, { label: 'card buttons' })).toBeNull(); + }); + it('renders all item labels and texts', () => { + const t = helpTrigger({ + title: 'The card buttons', + items: [ + { label: 'Copy', text: 'Copies the statement to your clipboard.' }, + { label: 'Load into console', text: 'Opens the statement in the console.' }, + { label: 'Run', text: 'Runs the statement. Elapsed time and rows show under the result.' }, + ], + }, { label: 'card buttons' }); + document.body.append(t); + t.click(); + const d = document.querySelector('.help-pop'); + expect(d.querySelector('.help-title').textContent).toBe('The card buttons'); + expect(d.textContent).toContain('Copy'); + expect(d.textContent).toContain('Copies the statement to your clipboard.'); + expect(d.textContent).toContain('Load into console'); + expect(d.textContent).toContain('Opens the statement in the console.'); + expect(d.textContent).toContain('Run'); + expect(d.textContent).toContain('Elapsed time and rows show under the result.'); + }); +}); + +describe('figure popover', () => { + beforeEach(() => { document.body.innerHTML = ''; }); + it('widens with .has-fig and wraps the figure (and an "open full size" link) to the full file', () => { + const t = helpTrigger({ why: 'x', figure: { file: 'model-curve.svg', caption: 'cap' } }, { label: 'y', patternId: '02-computed' }); + document.body.append(t); + t.click(); + const d = document.querySelector('.help-pop'); + expect(d.classList.contains('has-fig')).toBe(true); + const links = [...d.querySelectorAll('.help-fig a')]; + expect(links).toHaveLength(2); + links.forEach((a) => { + expect(a.getAttribute('href')).toBe('/figures/02-computed/model-curve.svg'); + expect(a.getAttribute('target')).toBe('_blank'); + expect(a.getAttribute('rel')).toBe('noopener'); + }); + expect(d.querySelector('.help-fig-open').textContent).toBe('Open full size ↗'); + }); +}); + +describe('hover tooltip', () => { + beforeEach(() => { document.body.innerHTML = ''; vi.useFakeTimers(); }); + afterEach(() => vi.useRealTimers()); + it('describes its trigger while open, and Esc anywhere closes it and drops the description', () => { + const t = helpTrigger({ why: 'Because.' }, { label: 'hover me' }); + document.body.append(t); + t.dispatchEvent(new Event('pointerenter')); vi.advanceTimersByTime(200); + const pop = document.querySelector('.help-pop'); + expect(pop.getAttribute('role')).toBe('tooltip'); + expect(pop.id).not.toBe(''); + expect(t.getAttribute('aria-describedby')).toBe(pop.id); + document.dispatchEvent(new KeyboardEvent('keydown', { key: 'Escape', bubbles: true })); + expect(document.querySelector('.help-pop')).toBeNull(); + expect(t.hasAttribute('aria-describedby')).toBe(false); + }); +}); + +describe('hover tooltip grace', () => { + beforeEach(() => { document.body.innerHTML = ''; vi.useFakeTimers(); }); + afterEach(() => { document.dispatchEvent(new KeyboardEvent('keydown', { key: 'Escape' })); vi.useRealTimers(); }); + const hover = () => { + const t = helpTrigger({ why: 'w', figure: { file: 'erd.svg', caption: 'c' } }, { label: 'x', patternId: '02-computed' }); + document.body.append(t); + t.dispatchEvent(new Event('pointerenter')); vi.advanceTimersByTime(250); + return t; + }; + it('survives the gap between the trigger and the popover', () => { + const t = hover(); + t.dispatchEvent(Object.assign(new Event('pointerleave'), { relatedTarget: null })); + vi.advanceTimersByTime(150); + const pop = document.querySelector('.help-pop'); + expect(pop).not.toBeNull(); + pop.dispatchEvent(new Event('pointerenter')); + vi.advanceTimersByTime(1000); + expect(document.querySelector('.help-pop')).not.toBeNull(); + expect(document.querySelector('.help-fig-open')).not.toBeNull(); + }); + it('closes shortly after the pointer leaves both trigger and popover', () => { + const t = hover(); + t.dispatchEvent(Object.assign(new Event('pointerleave'), { relatedTarget: null })); + vi.advanceTimersByTime(1000); + expect(document.querySelector('.help-pop')).toBeNull(); + const t2 = hover(); + const pop = document.querySelector('.help-pop'); + t2.dispatchEvent(Object.assign(new Event('pointerleave'), { relatedTarget: null })); + pop.dispatchEvent(new Event('pointerenter')); + pop.dispatchEvent(new Event('pointerleave')); + vi.advanceTimersByTime(1000); + expect(document.querySelector('.help-pop')).toBeNull(); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/measure.test.js b/labs/converged-modeling-patterns/app/test/unit/measure.test.js new file mode 100644 index 0000000..b414056 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/measure.test.js @@ -0,0 +1,72 @@ +// @vitest-environment jsdom +import { describe, it, expect } from 'vitest'; +import { renderMeasure } from '../../public/js/measure.js'; + +const P = { id: '02-computed', meta: { deck: '18–21', measure: { xLabel: 'CDR items', labX: 1000, deckSlides: '18–21', + calibration: [{ x: 10, ratio: 1.2 }, { x: 100, ratio: 5 }, { x: 1000, ratio: 33 }, { x: 5000, ratio: 260 }] } } }; +const side = (redo, blocks, kind = 'dml') => ({ sql: 'x', stats: { 'redo size': redo, 'db block changes': blocks, 'session logical reads': 10, 'CPU used by this session': 0 }, + result: kind === 'dml' ? { kind, rowsAffected: 1, elapsedMs: 2 } : { kind, elapsedMs: 2 } }); +// `.stat-name` also carries an ⓘ button once one is appended, so its textContent is the name plus +// "ⓘ" — match on the prefix rather than equality. +const statRow = (el, name) => [...el.querySelectorAll('.stat')].find((r) => r.querySelector('.stat-name').textContent.startsWith(name)); + +describe('renderMeasure', () => { + it('renders the curve with a live marker, per-row scaled bars with ratios, and explainers', () => { + const el = renderMeasure({ tag: 't', document: side(52160, 21), converged: side(1592, 11) }, { pattern: P }); + expect(el.querySelector('svg.curve circle.live-redo')).not.toBeNull(); + expect(el.querySelector('.stat-bars')).not.toBeNull(); + expect(el.textContent).toContain('32.8×'); + expect(el.querySelector('.what-happened')).not.toBeNull(); + expect(el.querySelector('.deck-thumb img').getAttribute('src')).toBe('/figures/02-computed/model-curve.svg'); + expect(el.textContent).not.toMatch(/NaN|Infinity/); + }); + + it('scales each stat row to its own max, not a scale shared across rows', () => { + // A 33x gap (52160 vs 1592) must read as a 33x gap, not the ~1.5x a shared log scale draws. + const el = renderMeasure({ tag: 't', document: side(52160, 21), converged: side(1592, 11) }, { pattern: P }); + const row = statRow(el, 'redo size'); + const coolBar = row.querySelector('.stat-bar.cool'); + expect(parseFloat(coolBar.style.width)).toBeLessThan(5); + }); + + it('an error side shows the error, no live marker and — for every ratio, even with stats on the error side', () => { + // The error side still carries stats (the failed statement did some work before it failed), + // so a ratio could be computed — it must not be: it would compare a partial write to a full one. + const err = { ...side(52160, 21), result: { kind: 'error', code: 'LAB-MEASURE', error: 'nope', elapsedMs: 0 } }; + const el = renderMeasure({ tag: 't', document: err, converged: side(1592, 11) }, { pattern: P }); + expect(el.querySelector('.result.error').textContent).toContain('LAB-MEASURE'); + expect(el.querySelector('svg.curve')).not.toBeNull(); + expect(el.querySelector('circle.live-redo')).toBeNull(); + const ratios = [...el.querySelectorAll('.stat-ratio')].map((r) => r.textContent); + expect(ratios).toEqual(['—', '—', '—', '—']); + expect(el.textContent).not.toMatch(/NaN|Infinity/); + }); + + it('shows — for every stat ratio when both sides error', () => { + const err = { sql: 'x', stats: {}, result: { kind: 'error', code: 'LAB-MEASURE', error: 'nope', elapsedMs: 0 } }; + const el = renderMeasure({ tag: 't', document: err, converged: err }, { pattern: P }); + const ratios = [...el.querySelectorAll('.stat-ratio')].map((r) => r.textContent); + expect(ratios).toEqual(['—', '—', '—', '—']); + expect(el.textContent).not.toMatch(/NaN|Infinity/); + }); + + it('shows exact stat values and the ratio for a specific run', () => { + const el = renderMeasure({ tag: 't', document: side(4752, 21, 'ok'), converged: side(864, 6) }, { pattern: P }); + const row = statRow(el, 'redo size'); + const vals = [...row.querySelectorAll('.stat-val')].map((v) => v.textContent); + expect(vals).toEqual(['4,752', '864']); + expect(row.querySelector('.stat-ratio').textContent).toBe('5.5×'); + }); + + it('a PL/SQL (kind ok) side shows — for rows affected', () => { + const el = renderMeasure({ tag: 't', document: side(4752, 21, 'ok'), converged: side(864, 6) }, { pattern: P }); + expect(el.querySelector('.rows-affected').textContent).toContain('rows affected — vs 1'); + }); + + it('a pattern without calibration still renders the bars and collapses to one column', () => { + const el = renderMeasure({ tag: 't', document: side(3060, 17), converged: side(500, 3) }, { pattern: { id: 'x', meta: { deck: '1', measure: null } } }); + expect(el.querySelector('svg.curve')).toBeNull(); + expect(el.querySelector('.stat-bars')).not.toBeNull(); + expect(el.querySelector('.measure-grid').classList.contains('no-curve')).toBe(true); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/mongo.test.js b/labs/converged-modeling-patterns/app/test/unit/mongo.test.js new file mode 100644 index 0000000..3d9feb2 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/mongo.test.js @@ -0,0 +1,83 @@ +import { describe, it, expect } from 'vitest'; +import { runMongo } from '../../src/db/mongo.js'; +import { parseMongoCommand } from '../../src/content/mongoCommand.js'; + +// The Oracle API for MongoDB does not honour maxTimeMS at all (verified against the +// live cmp-oracle container: a 1-second $sql aggregate and a listCollections both +// completed successfully with maxTimeMS: 1 passed straight through the driver — no +// timeout error). So the timing-based integration test the review suggested would be +// flaky-or-always-pass, not a real assertion. Instead these tests stub the driver and +// assert the *options object actually handed to it*: the system timeoutMs must always +// win over any maxTimeMS a user supplies in their command's options literal, on every +// path that accepts one (db.aggregate, collection aggregate, updateOne, updateMany, +// listCollections) — this is the actual line of defence for the single-permit gate. + +function fakeCursor(docs = []) { + return { + [Symbol.asyncIterator]() { + let i = 0; + return { next: async () => (i < docs.length ? { value: docs[i++], done: false } : { value: undefined, done: true }) }; + }, + close: async () => {}, + }; +} + +function fakeClient(capture) { + const coll = { + aggregate(pipeline, opts) { capture.collAggregate = opts; return fakeCursor(); }, + updateOne(filter, update, opts) { capture.updateOne = opts; return Promise.resolve({ matchedCount: 0, modifiedCount: 0 }); }, + updateMany(filter, update, opts) { capture.updateMany = opts; return Promise.resolve({ matchedCount: 0, modifiedCount: 0 }); }, + find() { return fakeCursor(); }, + findOne: async () => null, + countDocuments: async () => 0, + insertOne: async () => ({ acknowledged: true }), + insertMany: async () => ({ acknowledged: true }), + deleteOne: async () => ({ acknowledged: true }), + deleteMany: async () => ({ acknowledged: true }), + }; + const db = { + listCollections(filter, opts) { capture.listCollections = opts; return { toArray: async () => [] }; }, + aggregate(pipeline, opts) { capture.dbAggregate = opts; return fakeCursor(); }, + collection() { return coll; }, + }; + return { db: () => db }; +} + +const opts = { timeoutMs: 5, maxRows: 500, maxBytes: 1048576 }; + +describe('runMongo enforces the system maxTimeMS cap', () => { + it('forces maxTimeMS on db.aggregate, overriding a larger user-supplied value', async () => { + const capture = {}; + const client = fakeClient(capture); + await runMongo(client, 'CMP_USER', parseMongoCommand('db.aggregate([{ $sql: `select 1 from dual` }], { maxTimeMS: 600000 })'), opts); + expect(capture.dbAggregate.maxTimeMS).toBe(5); + }); + + it('forces maxTimeMS on collection aggregate, overriding a larger user-supplied value', async () => { + const capture = {}; + const client = fakeClient(capture); + await runMongo(client, 'CMP_USER', parseMongoCommand('db.lab_it_docs.aggregate([{$match:{}}], { maxTimeMS: 600000 })'), opts); + expect(capture.collAggregate.maxTimeMS).toBe(5); + }); + + it('forces maxTimeMS on updateOne, overriding a larger user-supplied value', async () => { + const capture = {}; + const client = fakeClient(capture); + await runMongo(client, 'CMP_USER', parseMongoCommand('db.lab_it_docs.updateOne({_id:"a"}, {$set:{n:1}}, { maxTimeMS: 600000 })'), opts); + expect(capture.updateOne.maxTimeMS).toBe(5); + }); + + it('forces maxTimeMS on updateMany, overriding a larger user-supplied value', async () => { + const capture = {}; + const client = fakeClient(capture); + await runMongo(client, 'CMP_USER', parseMongoCommand('db.lab_it_docs.updateMany({}, {$set:{n:1}}, { maxTimeMS: 600000 })'), opts); + expect(capture.updateMany.maxTimeMS).toBe(5); + }); + + it('sets maxTimeMS on show collections (listCollections)', async () => { + const capture = {}; + const client = fakeClient(capture); + await runMongo(client, 'CMP_USER', parseMongoCommand('show collections'), opts); + expect(capture.listCollections.maxTimeMS).toBe(5); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/mongoCommand.test.js b/labs/converged-modeling-patterns/app/test/unit/mongoCommand.test.js new file mode 100644 index 0000000..539407f --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/mongoCommand.test.js @@ -0,0 +1,44 @@ +import { describe, it, expect } from 'vitest'; +import { ObjectId } from 'mongodb'; +import { parseMongoCommand, MongoParseError } from '../../src/content/mongoCommand.js'; + +describe('parseMongoCommand', () => { + it('parses a collection findOne with an unquoted-key filter', () => { + expect(parseMongoCommand('db.xr_client_dv.findOne({_id: "C-001"});')).toEqual({ + kind: 'collection', collection: 'xr_client_dv', op: 'findOne', args: [{ _id: 'C-001' }], mods: {}, + }); + }); + it('parses db.aggregate with a multi-line $sql template literal and .toArray()', () => { + const c = parseMongoCommand('db.aggregate([{ $sql: `select machine_id\n from bk_sensor_readings` }]).toArray()'); + expect(c.kind).toBe('db'); + expect(c.op).toBe('aggregate'); + expect(c.args[0][0].$sql).toContain('from bk_sensor_readings'); + }); + it('collects sort/limit/skip modifiers on find', () => { + const c = parseMongoCommand('db.c.find({a:{$gt:-1}}).sort({a:-1}).skip(2).limit(5)'); + expect(c.args).toEqual([{ a: { $gt: -1 } }]); + expect(c.mods).toEqual({ sort: { a: -1 }, skip: 2, limit: 5 }); + }); + it('supports getCollection, regex, ISODate and ObjectId', () => { + const c = parseMongoCommand('db.getCollection("x").find({n:/^A/i, d:{$gte:ISODate("2026-08-01T00:00:00Z")}, o:ObjectId("65f000000000000000000001")})'); + expect(c.collection).toBe('x'); + expect(c.args[0].n).toBeInstanceOf(RegExp); + expect(c.args[0].d.$gte).toBeInstanceOf(Date); + expect(c.args[0].o).toBeInstanceOf(ObjectId); + }); + it('parses show collections', () => { + expect(parseMongoCommand('show collections')).toEqual({ kind: 'showCollections' }); + }); + it.each([ + ['db.c.drop()', /unsupported operation/], + ['db.c.find({a: `${x}`})', /template/], + ['db.c.find(process.exit())', /unsupported syntax/], + ['db.c.find({}); db.c.find({})', /one command/], + ['require("fs")', /db\./], + ['db.c.findOne({}).limit(1)', /only apply to find/], + ['db.c.find({...x})', /unsupported/], + ])('rejects %s', (text, msg) => { + expect(() => parseMongoCommand(text)).toThrow(MongoParseError); + expect(() => parseMongoCommand(text)).toThrow(msg); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/mongoScript.test.js b/labs/converged-modeling-patterns/app/test/unit/mongoScript.test.js new file mode 100644 index 0000000..ef755b9 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/mongoScript.test.js @@ -0,0 +1,49 @@ +import { describe, it, expect } from 'vitest'; +import fs from 'node:fs'; +import path from 'node:path'; +import { parseMongoScript } from '../../src/content/mongoScript.js'; + +describe('parseMongoScript', () => { + it('binds @step/@note to the next top-level db.* statement', () => { + const steps = parseMongoScript([ + 'function canon(x) { return x; }', + '// @step Read the projected document', + '// @note Same view, over the MongoDB API', + 'const d = db.xr_client_dv.findOne({_id: "C-001"});', + 'print(d);', + '// @step Hourly rollup with $sql', + 'const rows = db.aggregate([{ $sql: `select 1 from dual` }]).toArray();', + 'for (const r of rows) {', + ' // @step ignored inside a block', + ' print(r);', + '}', + ].join('\n')); + const noHelp = { why: null, look: null, figure: null }; + expect(steps).toEqual([ + { title: 'Read the projected document', notes: ['Same view, over the MongoDB API'], help: noHelp, command: 'db.xr_client_dv.findOne({_id: "C-001"})', line: 4 }, + { title: 'Hourly rollup with $sql', notes: [], help: noHelp, command: 'db.aggregate([{ $sql: `select 1 from dual` }])', line: 7 }, + ]); + }); + it('rejects @step on a statement that is not a db command', () => { + expect(() => parseMongoScript('// @step bad\nfor (const x of []) {}', 'f.js')).toThrow(/f\.js:2/); + }); + it('rejects @step on an alias call (must be db.*)', () => { + expect(() => parseMongoScript('const C = db.x;\n// @step bad\nC.findOne({});')).toThrow(); + }); + it('parses every pattern .js file', () => { + const root = path.resolve(import.meta.dirname, '../../../patterns'); + for (const dir of fs.readdirSync(root)) { + for (const f of fs.readdirSync(path.join(root, dir)).filter((n) => n.endsWith('.js'))) { + const p = path.join(root, dir, f); + expect(() => parseMongoScript(fs.readFileSync(p, 'utf8'), p)).not.toThrow(); + } + } + }); +}); + +describe('help annotations (mongo)', () => { + it('attaches // @why, @look and @figure to the step', () => { + const [s] = parseMongoScript('// @step Find\n// @why One document.\n// @look count is 1\n// @figure flow.svg The flow\ndb.c.find({})\n'); + expect(s.help).toEqual({ why: 'One document.', look: 'count is 1', figure: { file: 'flow.svg', caption: 'The flow' } }); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/patterns.test.js b/labs/converged-modeling-patterns/app/test/unit/patterns.test.js new file mode 100644 index 0000000..99aa18f --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/patterns.test.js @@ -0,0 +1,143 @@ +import { describe, it, expect } from 'vitest'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { loadPatterns } from '../../src/content/patterns.js'; + +const FM = `--- +title: T +industry: I +deck: "14–17" +problem: P +knobs: + - { name: Diversity, setting: High } + - { name: Read / write, setting: Reads } + - { name: Update locality, setting: Fans out, hot: true } +--- +# body +`; + +function fixture(files) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'pat-')); + const dir = path.join(root, '09-demo'); + fs.mkdirSync(dir); + for (const [n, t] of Object.entries(files)) fs.writeFileSync(path.join(dir, n), t); + return root; +} + +const DOC = 'CREATE TABLE d (a NUMBER);\n-- @step Read\nSELECT a FROM d;\n-- @measure m\nUPDATE d SET a = 1;\n'; +const CONV = 'CREATE TABLE c (a NUMBER);\n-- @step Read\nSELECT a FROM c;\n-- @measure m\nUPDATE c SET a = 1;\n'; + +describe('loadPatterns (fixtures)', () => { + it('builds cards, setup, measure pairs and a version', () => { + const [p] = loadPatterns(fixture({ 'README.md': FM, '01-document-model.sql': DOC, '02-converged.sql': CONV })); + expect(p.id).toBe('09-demo'); + expect(p.meta.knobs[2]).toEqual({ name: 'Update locality', setting: 'Fans out', hot: true, help: null }); + expect(p.lanes.document.map((s) => s.title)).toEqual(['Read']); + expect(p.setup.converged.map((s) => s.sql)).toEqual(['CREATE TABLE c (a NUMBER)']); + expect(p.measures).toHaveLength(1); + expect(p.measures[0].document.sql).toBe('UPDATE d SET a = 1'); + expect(p.version).toMatch(/^[0-9a-f]{16}$/); + expect(p.lanes.mongo).toEqual([]); + }); + it('rejects a missing front matter block', () => { + expect(() => loadPatterns(fixture({ 'README.md': '# no fm', '01-document-model.sql': DOC, '02-converged.sql': CONV }))).toThrow(/front matter/); + }); + it('rejects a one-sided measure tag', () => { + expect(() => loadPatterns(fixture({ 'README.md': FM, '01-document-model.sql': DOC, '02-converged.sql': CONV.replace('@measure m', '@step Upd') }))).toThrow(/measure "m"/); + }); + it('rejects a measure statement that commits', () => { + const bad = DOC.replace('UPDATE d SET a = 1;', 'BEGIN UPDATE d SET a = 1; COMMIT; END;\n/'); + expect(() => loadPatterns(fixture({ 'README.md': FM, '01-document-model.sql': bad, '02-converged.sql': CONV }))).toThrow(/COMMIT/); + }); +}); + +describe('loadPatterns (the real lab)', () => { + const patterns = loadPatterns(path.resolve(import.meta.dirname, '../../../patterns')); + it('loads all six patterns with metadata', () => { + expect(patterns.map((p) => p.id)).toEqual(['01-extended-reference', '02-computed', '03-bucket', '04-subset', '05-tree-hierarchy', '06-outlier']); + for (const p of patterns) { + expect(p.meta.title && p.meta.industry && p.meta.problem && p.meta.deck, p.id).toBeTruthy(); + expect(p.meta.knobs.map((k) => k.hot), p.id).toEqual([false, false, true]); + expect(p.lanes.document.length, p.id).toBeGreaterThanOrEqual(2); + expect(p.lanes.converged.length, p.id).toBeGreaterThanOrEqual(2); + expect(p.measures.length, p.id).toBeGreaterThanOrEqual(1); + } + }); + it('has MongoDB cards where the pattern has a Mongo lane', () => { + const byId = Object.fromEntries(patterns.map((p) => [p.id, p])); + for (const id of ['01-extended-reference', '02-computed', '03-bucket', '06-outlier']) expect(byId[id].lanes.mongo.length, id).toBeGreaterThanOrEqual(1); + }); + it('every referenced figure, and each pattern\'s model-curve.svg, exists under patterns//figures/', () => { + const root = path.resolve(import.meta.dirname, '../../../patterns'); + let checked = 0; + for (const p of patterns) { + const cards = [...p.lanes.document, ...p.lanes.converged, ...p.lanes.mongo]; + const files = new Set(['model-curve.svg']); + for (const c of cards) if (c.help?.figure?.file) files.add(c.help.figure.file); + for (const t of Object.values(p.meta.help.tabs)) if (t?.figure?.file) files.add(t.figure.file); + for (const f of files) { + expect(fs.existsSync(path.join(root, p.id, 'figures', f)), `${p.id}/figures/${f}`).toBe(true); + checked++; + } + } + expect(checked).toBeGreaterThan(patterns.length); // @figure references were actually found + }); + it('is deterministic', () => { + const again = loadPatterns(path.resolve(import.meta.dirname, '../../../patterns')); + expect(again.map((p) => p.version)).toEqual(patterns.map((p) => p.version)); + }); +}); + +const FM_HELP = `--- +title: T +industry: I +deck: "14–17" +problem: P +knobs: + - { name: Diversity, setting: High, help: "Four consumers read it." } + - { name: Read / write, setting: Reads } + - { name: Update locality, setting: Fans out, hot: true } +help: + tabs: + document: { why: "The starting point.", look: "One read.", figure: "doc-shape.svg" } + measure: { why: "Same write, both models." } +measure: + x_label: "CDR line items in the document" + lab_x: 1000 + deck_slides: "18–21" + calibration: + - { x: 10, ratio: 1.2 } + - { x: 100, ratio: 5 } + - { x: 1000, ratio: 33 } + - { x: 5000, ratio: 260 } +--- +# body +`; + +describe('help and measure front matter', () => { + it('parses tab help, knob help and measure calibration', () => { + const [p] = loadPatterns(fixture({ 'README.md': FM_HELP, '01-document-model.sql': DOC, '02-converged.sql': CONV })); + expect(p.meta.knobs[0].help).toBe('Four consumers read it.'); + expect(p.meta.knobs[1].help).toBeNull(); + expect(p.meta.help.tabs.document).toEqual({ why: 'The starting point.', look: 'One read.', figure: { file: 'doc-shape.svg', caption: null } }); + expect(p.meta.help.tabs.converged).toBeUndefined(); + expect(p.meta.measure).toEqual({ xLabel: 'CDR line items in the document', labX: 1000, deckSlides: '18–21', + calibration: [{ x: 10, ratio: 1.2 }, { x: 100, ratio: 5 }, { x: 1000, ratio: 33 }, { x: 5000, ratio: 260 }] }); + }); + it('absent help and measure give empty structures, not errors', () => { + const [p] = loadPatterns(fixture({ 'README.md': FM, '01-document-model.sql': DOC, '02-converged.sql': CONV })); + expect(p.meta.help).toEqual({ tabs: {} }); + expect(p.meta.measure).toBeNull(); + expect(p.meta.knobs.every((k) => k.help === null)).toBe(true); + }); + it('rejects a measure block without a non-empty x_label', () => { + for (const bad of [FM_HELP.replace(' x_label: "CDR line items in the document"\n', ''), FM_HELP.replace('x_label: "CDR line items in the document"', 'x_label: ""')]) { + expect(() => loadPatterns(fixture({ 'README.md': bad, '01-document-model.sql': DOC, '02-converged.sql': CONV }))).toThrow(/x_label/); + } + }); + it('rejects a calibration with fewer than 4 points or lab_x outside the range', () => { + const bad = FM_HELP.replace('lab_x: 1000', 'lab_x: 99999'); + expect(() => loadPatterns(fixture({ 'README.md': bad, '01-document-model.sql': DOC, '02-converged.sql': CONV }))).toThrow(/lab_x/); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/reaper.test.js b/labs/converged-modeling-patterns/app/test/unit/reaper.test.js new file mode 100644 index 0000000..e24e715 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/reaper.test.js @@ -0,0 +1,110 @@ +import { describe, it, expect, vi, afterEach } from 'vitest'; +import { Workspaces } from '../../src/services/workspaces.js'; + +const deferred = () => { let resolve; const p = new Promise((a) => { resolve = a; }); return { p, resolve }; }; +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + +afterEach(() => vi.restoreAllMocks()); + +describe('Workspaces.startReaper', () => { + it('skips a tick while the previous pass is still in flight, and resumes after it finishes', async () => { + const ws = new Workspaces({ cfg: {}, pools: {} }); + vi.spyOn(ws, 'pendingCount').mockResolvedValue(1); + const reap = vi.spyOn(ws, 'reapPending').mockResolvedValue({ dropped: 0, pending: 0 }); + const hold = deferred(); + const runPass = vi.fn(async (fn) => { const out = await fn(); await hold.p; return out; }); + + const stop = ws.startReaper(20, runPass); + try { + await sleep(150); // well over 2 intervals while the first pass is blocked + expect(runPass).toHaveBeenCalledTimes(1); + expect(reap).toHaveBeenCalledTimes(1); + + hold.resolve(); + await sleep(100); + expect(reap.mock.calls.length).toBeGreaterThan(1); // the in-flight flag was cleared + } finally { + stop(); + } + }); + + it('logs a failed pass (e.g. a busy gate) as one line and retries on the next tick', async () => { + const ws = new Workspaces({ cfg: {}, pools: {} }); + vi.spyOn(ws, 'pendingCount').mockResolvedValue(1); + vi.spyOn(ws, 'reapPending').mockResolvedValue({ dropped: 0, pending: 0 }); + const err = vi.spyOn(console, 'error').mockImplementation(() => {}); + const runPass = vi.fn() + .mockRejectedValueOnce(Object.assign(new Error('busy, retry'), { status: 429 })) + .mockImplementation((fn) => fn()); + + const stop = ws.startReaper(20, runPass); + try { + await sleep(100); + expect(err).toHaveBeenCalledWith('[lab-ui] reapPending failed: busy, retry'); + expect(runPass.mock.calls.length).toBeGreaterThan(1); + } finally { + stop(); + } + }); +}); + +// A stand-in control pool: `handler(sql, binds)` returns the execute() result (default: no rows). +function fakeControl(handler = () => undefined) { + const calls = []; + const conn = { + execute: vi.fn(async (sql, binds) => { calls.push(sql); return (await handler(sql, binds)) ?? { rows: [] }; }), + commit: vi.fn(async () => {}), + close: vi.fn(async () => {}), + }; + return { pools: { control: { getConnection: async () => conn } }, calls }; +} + +describe('reaper gate use (M2)', () => { + it('checks lab_pending_drop first and takes the gate only when there is something to reap', async () => { + const { pools, calls } = fakeControl((sql) => (/COUNT\(\*\)/.test(sql) ? { rows: [{ N: 0 }] } : undefined)); + const ws = new Workspaces({ cfg: {}, pools }); + const runPass = vi.fn((fn) => fn()); + const stop = ws.startReaper(20, runPass); + try { + await sleep(100); + expect(calls.some((s) => /SELECT COUNT\(\*\) AS n FROM lab_pending_drop/.test(s))).toBe(true); + expect(runPass).not.toHaveBeenCalled(); + } finally { + stop(); + } + }); +}); + +describe('Workspaces.dropAll (M3 + name check)', () => { + it('locks and parks WS_ workspaces, skips a non-WS_ name, and returns without reaping', async () => { + const log = vi.spyOn(console, 'warn').mockImplementation(() => {}); + const { pools, calls } = fakeControl(); + const ws = new Workspaces({ cfg: {}, pools }); + vi.spyOn(ws, 'list').mockResolvedValue([{ schema: 'WS_ABC123' }, { schema: 'CMP_USER; DROP' }]); + const reap = vi.spyOn(ws, 'reapPending'); + const drop = vi.fn(async () => {}); + expect(await ws.dropAll({ drop })).toEqual({ dropped: 0, pending: 1 }); + expect(reap).not.toHaveBeenCalled(); + expect(calls.filter((s) => /ACCOUNT LOCK/.test(s))).toEqual(['ALTER USER WS_ABC123 ACCOUNT LOCK']); + expect(calls.some((s) => /CMP_USER/.test(s))).toBe(false); + expect(log).toHaveBeenCalledWith(expect.stringMatching(/skipped "CMP_USER; DROP": not a WS_ workspace name/)); + }); +}); + +describe('Workspaces.reapPending session kill', () => { + it('ignores KILLED sessions and tolerates ORA-00031 while killing', async () => { + let drops = 0; + const { pools, calls } = fakeControl((sql) => { + if (/FROM lab_pending_drop/.test(sql) && /parked_sec/.test(sql)) return { rows: [{ SCHEMA_NAME: 'WS_ABC123', PARKED_SEC: 300 }] }; + if (/^DROP USER/.test(sql)) { drops += 1; if (drops === 1) throw Object.assign(new Error('ORA-01940'), { errorNum: 1940 }); return undefined; } + if (/SELECT sid/.test(sql)) return { rows: [{ SID: 10, SERIAL: 20 }] }; + if (/KILL SESSION/.test(sql)) throw Object.assign(new Error('ORA-00031: session marked for kill'), { errorNum: 31 }); + if (/COUNT\(\*\) AS n FROM v\$session/.test(sql)) return { rows: [{ N: 0 }] }; + return undefined; + }); + vi.spyOn(console, 'log').mockImplementation(() => {}); + const ws = new Workspaces({ cfg: {}, pools }); + expect(await ws.reapPending()).toEqual({ dropped: 1, pending: 0 }); + expect(calls.find((s) => /SELECT sid/.test(s))).toMatch(/status <> 'KILLED'/); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/results.test.js b/labs/converged-modeling-patterns/app/test/unit/results.test.js new file mode 100644 index 0000000..8a2ae59 --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/results.test.js @@ -0,0 +1,58 @@ +// app/test/unit/results.test.js +// @vitest-environment jsdom +import { describe, it, expect } from 'vitest'; +import { renderResult, renderRun } from '../../public/js/results.js'; + +describe('renderResult', () => { + it('renders rows as a table with escaped values', () => { + const el = renderResult({ kind: 'rows', columns: ['NAME'], rows: [{ NAME: '' }], rowCount: 1, truncated: false, elapsedMs: 3 }); + expect(el.querySelector('table')).not.toBeNull(); + expect(el.querySelector('img')).toBeNull(); + expect(el.querySelector('td').textContent).toBe(''); + }); + it('renders JSON-valued cells and documents as a collapsible tree', () => { + const el = renderResult({ kind: 'docs', docs: [{ _id: 'C-001', advisor: { office: 'NYC-09' } }], count: 1, truncated: false, elapsedMs: 2 }); + expect(el.querySelectorAll('details').length).toBeGreaterThan(0); + expect(el.textContent).toContain('NYC-09'); + }); + it('renders DML, DDL, counts, collections and writes as messages', () => { + expect(renderResult({ kind: 'dml', rowsAffected: 3, elapsedMs: 1 }).textContent).toContain('3 rows affected'); + expect(renderResult({ kind: 'ok', elapsedMs: 1 }).textContent).toContain('Statement executed'); + expect(renderResult({ kind: 'count', count: 7, elapsedMs: 1 }).textContent).toContain('7'); + expect(renderResult({ kind: 'collections', names: ['a', 'b'], elapsedMs: 1 }).textContent).toContain('a'); + expect(renderResult({ kind: 'write', result: { modifiedCount: 1 }, elapsedMs: 1 }).textContent).toContain('modifiedCount'); + }); + it('renders errors with their code', () => { + const el = renderResult({ kind: 'error', code: 'ORA-00942', error: 'ORA-00942: table or view does not exist', elapsedMs: 1 }); + expect(el.classList.contains('error')).toBe(true); + expect(el.textContent).toContain('ORA-00942'); + }); + it('flags truncation', () => { + expect(renderResult({ kind: 'rows', columns: ['N'], rows: [{ N: 1 }], rowCount: 1, truncated: true, elapsedMs: 1 }).textContent) + .toMatch(/truncated/i); + }); + it('formats an ISO timestamp cell as YYYY-MM-DD HH:MM:SS', () => { + const el = renderResult({ kind: 'rows', columns: ['HOUR_START'], rows: [{ HOUR_START: '2026-01-10T09:00:00.000Z' }], rowCount: 1, truncated: false, elapsedMs: 1 }); + expect(el.querySelector('td').textContent).toBe('2026-01-10 09:00:00'); + }); + it('keeps a non-zero fraction on a formatted timestamp cell', () => { + const el = renderResult({ kind: 'rows', columns: ['CLAIM_TS'], rows: [{ CLAIM_TS: '2026-01-10T09:00:00.123Z' }], rowCount: 1, truncated: false, elapsedMs: 1 }); + expect(el.querySelector('td').textContent).toBe('2026-01-10 09:00:00.123'); + }); + it('leaves a non-timestamp string untouched', () => { + const el = renderResult({ kind: 'rows', columns: ['NOTE'], rows: [{ NOTE: '2026-01-10 note' }], rowCount: 1, truncated: false, elapsedMs: 1 }); + expect(el.querySelector('td').textContent).toBe('2026-01-10 note'); + }); + it('formats a timestamp inside a docs tree the same way', () => { + const el = renderResult({ kind: 'docs', docs: [{ _id: 'C-001', claim_ts: '2026-01-10T09:00:00.000Z' }], count: 1, truncated: false, elapsedMs: 1 }); + expect(el.textContent).toContain('2026-01-10 09:00:00'); + expect(el.textContent).not.toContain('2026-01-10T09:00:00.000Z'); + }); +}); + +describe('renderRun', () => { + it('shows the cache badge', () => { + const el = renderRun({ lane: 'sql', cached: true, results: [{ kind: 'ok', elapsedMs: 0 }] }); + expect(el.textContent).toContain('served from cache'); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/runnerCache.test.js b/labs/converged-modeling-patterns/app/test/unit/runnerCache.test.js new file mode 100644 index 0000000..d7d83ff --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/runnerCache.test.js @@ -0,0 +1,61 @@ +// Runner cache eligibility (I1), against a stub connection — no database needed. +import { describe, it, expect, beforeEach } from 'vitest'; +import path from 'node:path'; +import { loadConfig } from '../../src/config.js'; +import { Gate } from '../../src/gate.js'; +import { ResultCache } from '../../src/cache.js'; +import { loadPatterns } from '../../src/content/patterns.js'; +import { Runner } from '../../src/services/runner.js'; + +const patterns = loadPatterns(path.resolve(import.meta.dirname, '../../../patterns')); +const p = patterns.find((x) => x.id === '01-extended-reference'); +const readCard = p.lanes.converged.find((s) => /^\s*(SELECT|WITH)\b/i.test(s.sql)); +const user = { id: 'solo', workspace: { schema: 'CMP_USER', password: 'x' } }; +let runner; let executed; let cache; + +beforeEach(() => { + executed = []; + const conn = { + execute: async (sql) => { executed.push(sql); return { metaData: [{ name: 'X' }], rows: [{ X: 1 }] }; }, + close: async () => {}, + break: async () => {}, + }; + const workspaces = { + builtVersion: (s, id) => patterns.find((x) => x.id === id).version, + isDirty: () => false, + markDirty: async () => {}, + }; + const cfg = loadConfig({}); + cache = new ResultCache(); + runner = new Runner({ cfg, gate: new Gate(cfg.gate), cache, pools: { exec: { getConnection: async () => conn } }, mongo: {}, workspaces, patterns }); +}); + +const run = (text) => runner.runSql({ user, patternId: p.id, text }); + +describe('result cache eligibility', () => { + it('never caches ad hoc console text', async () => { + expect((await run('SELECT USER FROM dual')).cached).toBe(false); + expect((await run('SELECT USER FROM dual')).cached).toBe(false); + expect(executed).toHaveLength(2); + expect(cache.size).toBe(0); + }); + + it('does not cache comment-only text', async () => { + expect((await run('--x')).cached).toBe(false); + expect((await run('--y')).cached).toBe(false); + expect(cache.size).toBe(0); + }); + + it('caches a card SELECT run exactly as shown', async () => { + expect((await run(readCard.sql)).cached).toBe(false); + expect((await run(readCard.sql)).cached).toBe(true); + expect(executed).toHaveLength(1); + }); + + it('still hits after a leading/trailing whitespace edit, but not after any other edit', async () => { + await run(readCard.sql); + expect((await run(` ${readCard.sql} \n`)).cached).toBe(true); + expect((await run(`${readCard.sql} -- mine`)).cached).toBe(false); + expect((await run(readCard.sql.replace(/ /, ' '))).cached).toBe(false); + }); +}); diff --git a/labs/converged-modeling-patterns/app/test/unit/sqlParser.test.js b/labs/converged-modeling-patterns/app/test/unit/sqlParser.test.js new file mode 100644 index 0000000..7f23fae --- /dev/null +++ b/labs/converged-modeling-patterns/app/test/unit/sqlParser.test.js @@ -0,0 +1,121 @@ +import { describe, it, expect } from 'vitest'; +import fs from 'node:fs'; +import path from 'node:path'; +import { parseSqlFile, isSetup, splitConsoleSql } from '../../src/content/sqlParser.js'; + +describe('parseSqlFile', () => { + it('attaches @step/@note/@measure to the next statement and keeps setup untagged', () => { + const s = parseSqlFile([ + '-- plain comment', + 'DROP TABLE t PURGE;', + '-- @step Read it', + '-- @note first note', + '-- @note second', + 'SELECT a', + 'FROM t;', + '-- @measure move', + 'UPDATE t SET a = 1;', + ].join('\n')); + expect(s.map((x) => x.sql)).toEqual(['DROP TABLE t PURGE', 'SELECT a\nFROM t', 'UPDATE t SET a = 1']); + expect(s[1]).toMatchObject({ title: 'Read it', notes: ['first note', 'second'], measure: null, line: 6 }); + expect(s[2]).toMatchObject({ title: null, measure: 'move' }); + expect(s.map(isSetup)).toEqual([true, false, false]); + }); + + it('treats a BEGIN block ended by "/" as one PL/SQL statement', () => { + const s = parseSqlFile('BEGIN\n FOR r IN (SELECT 1 x FROM dual) LOOP\n NULL;\n END LOOP;\nEND;\n/\nSELECT 1 FROM dual;'); + expect(s).toHaveLength(2); + expect(s[0].plsql).toBe(true); + expect(s[0].sql.endsWith('END;')).toBe(true); + expect(s[1].sql).toBe('SELECT 1 FROM dual'); + }); + + it('treats CREATE OR REPLACE TRIGGER as PL/SQL', () => { + const s = parseSqlFile('CREATE OR REPLACE TRIGGER trg AFTER INSERT ON t FOR EACH ROW\nBEGIN\n UPDATE s SET n = n + 1;\nEND;\n/'); + expect(s).toHaveLength(1); + expect(s[0].plsql).toBe(true); + }); + + it('does not split or strip inside string literals (JSON seed data)', () => { + const s = parseSqlFile(`INSERT INTO t VALUES ('{"a":"x;y","b":"--z","c":"it''s"}');\nSELECT 1 FROM dual;`); + expect(s).toHaveLength(2); + expect(s[0].sql).toContain('x;y'); + expect(s[0].sql).toContain('--z'); + expect(s[0].sql).toContain("it''s"); + }); + + it('drops sqlplus directives only at statement start', () => { + const s = parseSqlFile('SET HEADING OFF PAGESIZE 0\nSELECT 1 FROM dual;\nUPDATE t\nSET data = JSON_TRANSFORM(data, SET \'$.a\' = 1)\nWHERE id = 1;'); + expect(s.map((x) => x.sql)).toEqual(['SELECT 1 FROM dual', "UPDATE t\nSET data = JSON_TRANSFORM(data, SET '$.a' = 1)\nWHERE id = 1"]); + }); + + it('ignores a trailing comment after the semicolon', () => { + expect(parseSqlFile('SELECT 1 FROM dual; -- one')[0].sql).toBe('SELECT 1 FROM dual'); + }); + + it('parses every pattern SQL file into clean statements', () => { + const root = path.resolve(import.meta.dirname, '../../../patterns'); + for (const dir of fs.readdirSync(root)) { + for (const f of ['01-document-model.sql', '02-converged.sql']) { + const p = path.join(root, dir, f); + if (!fs.existsSync(p)) continue; + const stmts = parseSqlFile(fs.readFileSync(p, 'utf8')); + expect(stmts.length, p).toBeGreaterThan(0); + for (const st of stmts) { + expect(st.sql.trim(), p).not.toBe(''); + expect(st.sql, p).not.toMatch(/\n\/\s*$/); + expect(st.sql, p).not.toMatch(/^(SET|PROMPT|WHENEVER)\s/); + } + } + } + }); +}); + +describe('splitConsoleSql', () => { + it('accepts a trailing semicolon or sqlplus slash', () => { + expect(splitConsoleSql('SELECT 1 FROM dual;')).toEqual(['SELECT 1 FROM dual']); + expect(splitConsoleSql('SELECT 1 FROM dual\n/')).toEqual(['SELECT 1 FROM dual']); + }); + it('splits several statements', () => { + expect(splitConsoleSql('SELECT 1 FROM dual;\nSELECT 2 FROM dual;')).toHaveLength(2); + }); + it('keeps a PL/SQL block without a closing slash intact', () => { + expect(splitConsoleSql('BEGIN NULL; END;')).toEqual(['BEGIN NULL; END;']); + }); + it('splits several statements on one line', () => { + expect(splitConsoleSql('SELECT 1 FROM dual; SELECT 2 FROM dual;')).toEqual(['SELECT 1 FROM dual', 'SELECT 2 FROM dual']); + expect(splitConsoleSql("SELECT ';' AS s FROM dual; SELECT 2 FROM dual -- two\n;")).toEqual(["SELECT ';' AS s FROM dual", 'SELECT 2 FROM dual -- two']); + }); + it('keeps a trailing comment after the last semicolon out of the statements', () => { + expect(splitConsoleSql('SELECT 1 FROM dual; -- done')).toEqual(['SELECT 1 FROM dual']); + }); + it('treats SET TRANSACTION and SET ROLE as SQL, not SQL*Plus', () => { + expect(splitConsoleSql('SET TRANSACTION READ ONLY;\nSELECT 1 FROM dual;')).toEqual(['SET TRANSACTION READ ONLY', 'SELECT 1 FROM dual']); + expect(splitConsoleSql('set role none;')).toEqual(['set role none']); + expect(splitConsoleSql('SET PAGESIZE 50\nSELECT 1 FROM dual;')).toEqual(['SELECT 1 FROM dual']); + }); + it('returns [] for blank input', () => { + expect(splitConsoleSql(' \n -- only a comment\n')).toEqual([]); + }); +}); + +describe('help annotations', () => { + it('attaches @why, @look and @figure to the next statement', () => { + const [s] = parseSqlFile([ + '-- @step Read it', + '-- @why Because the read is one document.', + '-- @look The row count stays at 1.', + '-- @figure doc-shape.svg The document as built', + 'SELECT 1 FROM dual;', + ].join('\n')); + expect(s.help).toEqual({ why: 'Because the read is one document.', look: 'The row count stays at 1.', figure: { file: 'doc-shape.svg', caption: 'The document as built' } }); + }); + it('defaults help to nulls and joins repeated @why lines', () => { + const [a, b] = parseSqlFile('-- @step A\nSELECT 1 FROM dual;\n-- @step B\n-- @why one\n-- @why two\nSELECT 2 FROM dual;'); + expect(a.help).toEqual({ why: null, look: null, figure: null }); + expect(b.help.why).toBe('one two'); + }); + it('rejects a @figure file name that is not a plain .svg', () => { + expect(() => parseSqlFile('-- @step A\n-- @figure ../x.svg c\nSELECT 1 FROM dual;')).toThrow(/@figure/); + }); +}); diff --git a/labs/converged-modeling-patterns/app/vitest.config.js b/labs/converged-modeling-patterns/app/vitest.config.js new file mode 100644 index 0000000..09c4250 --- /dev/null +++ b/labs/converged-modeling-patterns/app/vitest.config.js @@ -0,0 +1,9 @@ +import { defineConfig } from 'vitest/config'; +export default defineConfig({ + test: { + environment: 'node', + testTimeout: 20000, + hookTimeout: 120000, + fileParallelism: false, // integration tests share one database + }, +}); diff --git a/labs/converged-modeling-patterns/compose.yml b/labs/converged-modeling-patterns/compose.yml new file mode 100644 index 0000000..864ba97 --- /dev/null +++ b/labs/converged-modeling-patterns/compose.yml @@ -0,0 +1,80 @@ +name: converged-modeling-patterns + +# One service: Oracle AI Database 26ai Free + ORDS + the Oracle API for MongoDB, +# built fresh from ./docker (see docker/Dockerfile). Exposes SQL (1521), ORDS +# (8181), and the Mongo wire protocol (27017). Host ports default to NON-standard +# values so this never collides with another Oracle/ORDS/Mongo stack on the host; +# run.sh drives everything in-container, so the host mapping is only for your own +# clients. Override with CMP_PORT / CMP_ORDS_PORT / CMP_MONGO_PORT. +# +# Every host port binds to 127.0.0.1 by default. The database ports stay there even in +# event mode: publishing them to the room would hand attendees LAB_ADMIN and SYS. +# Only the console is opened up for an event (CMP_UI_BIND=0.0.0.0). +services: + oracle: + build: + context: ./docker + dockerfile: Dockerfile + image: converged-modeling-patterns-oracle:latest + container_name: cmp-oracle + ports: + - '${CMP_DB_BIND:-127.0.0.1}:${CMP_PORT:-1522}:1521' # SQL*Net + - '${CMP_DB_BIND:-127.0.0.1}:${CMP_ORDS_PORT:-8182}:8181' # ORDS / Database Actions + - '${CMP_DB_BIND:-127.0.0.1}:${CMP_MONGO_PORT:-27018}:27017' # Oracle API for MongoDB + environment: + ORACLE_PASSWORD: '${ORACLE_PASSWORD:-Sandbox2026}' # SYS / SYSTEM + APP_USER: 'CMP_USER' + APP_USER_PASSWORD: '${CMP_PASSWORD:-CmpUser2026}' + healthcheck: + # DB readiness only; run.sh polls ORDS + Mongo itself before the Mongo lane. + test: ['CMD', 'healthcheck.sh'] + interval: 10s + timeout: 10s + retries: 60 + start_period: 180s + volumes: + - cmp-data:/opt/oracle/oradata + + lab-ui: + build: + context: . + dockerfile: app/Dockerfile + target: runtime + image: converged-modeling-patterns-lab-ui:latest + container_name: cmp-lab-ui + depends_on: + oracle: + condition: service_healthy + ports: + - '${CMP_UI_BIND:-127.0.0.1}:${CMP_UI_PORT:-3100}:3000' # hands-on console; event mode: CMP_UI_BIND=0.0.0.0 + environment: + LAB_MODE: '${LAB_MODE:-solo}' # solo | event + ADMIN_PASSWORD: '${ADMIN_PASSWORD:-}' # required when LAB_MODE=event + EVENT_CODE: '${EVENT_CODE:-}' + ORACLE_PASSWORD: '${ORACLE_PASSWORD:-Sandbox2026}' + CMP_PASSWORD: '${CMP_PASSWORD:-CmpUser2026}' + LAB_ADMIN_PASSWORD: '${LAB_ADMIN_PASSWORD:-LabAdmin2026}' + DB_POOL_MAX: '${DB_POOL_MAX:-1}' + MONGO_POOL_MAX: '${MONGO_POOL_MAX:-1}' + + lab-ui-test: + profiles: ['test'] + build: + context: . + dockerfile: app/Dockerfile + target: test + depends_on: + oracle: + condition: service_healthy + environment: + DB_HOST: oracle + DB_PORT: '1521' + MONGO_HOST: oracle + MONGO_PORT: '27017' + ORACLE_PASSWORD: '${ORACLE_PASSWORD:-Sandbox2026}' + CMP_PASSWORD: '${CMP_PASSWORD:-CmpUser2026}' + # Same value lab-ui uses: the tests' bootstrap re-sets LAB_ADMIN's password to this. + LAB_ADMIN_PASSWORD: '${LAB_ADMIN_PASSWORD:-LabAdmin2026}' + +volumes: + cmp-data: diff --git a/labs/converged-modeling-patterns/docker/Dockerfile b/labs/converged-modeling-patterns/docker/Dockerfile new file mode 100644 index 0000000..241bdc4 --- /dev/null +++ b/labs/converged-modeling-patterns/docker/Dockerfile @@ -0,0 +1,36 @@ +# Oracle AI Database 26ai Free + ORDS + Oracle API for MongoDB. +# +# Built fresh for this repo — NOT copied from another lab. Two deliberate choices: +# * newest gvenzl faststart release line: 23.26.3 (year.quarter numbering; the +# 23.26.x line IS the 26ai release). Confirmed newest on Docker Hub at build time. +# * NO vector/ONNX model layer — these patterns use no vectors, so that ~130 MB +# download is skipped entirely. This image is DB + ORDS + mongosh, nothing more. +FROM gvenzl/oracle-free:23.26.3-faststart + +USER root + +# --- ORDS (latest) ----------------------------------------------------------- +# Installs Oracle REST Data Services, which fronts both the Database API and the +# Oracle API for MongoDB (port 27017). +COPY scripts/install-ords.sh /tmp/install-ords.sh +RUN chmod +x /tmp/install-ords.sh && /tmp/install-ords.sh && rm /tmp/install-ords.sh +RUN chown -R oracle:oinstall /etc/ords + +# --- mongosh (current MongoDB 8.0 repo) -------------------------------------- +# The Mongo lane of run.sh drives *.js scripts through mongosh, in-container. +# $basearch lets dnf pick x86_64 or aarch64, so the image also builds on Apple silicon. +RUN printf '[mongodb-org-8.0]\nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/redhat/8/mongodb-org/8.0/$basearch/\ngpgcheck=1\nenabled=1\ngpgkey=https://pgp.mongodb.com/server-8.0.asc\n' > /etc/yum.repos.d/mongodb-org-8.0.repo \ + && microdnf install -y mongodb-mongosh \ + && microdnf clean all + +# --- schema init + custom entrypoint ----------------------------------------- +# initdb scripts grant CMP_USER the privileges the patterns use. The entrypoint +# installs ORDS, ORDS-enables CMP_USER, and turns the Mongo API on (TLS off for +# local dev), then starts both Oracle and ORDS. +COPY init/ /container-entrypoint-initdb.d/ +COPY scripts/entrypoint.sh /opt/oracle/scripts/entrypoint.sh +RUN chmod +x /opt/oracle/scripts/entrypoint.sh + +USER oracle +EXPOSE 1521 8181 27017 +ENTRYPOINT ["/opt/oracle/scripts/entrypoint.sh"] diff --git a/labs/converged-modeling-patterns/docker/init/01-grants.sql b/labs/converged-modeling-patterns/docker/init/01-grants.sql new file mode 100644 index 0000000..66fb4a9 --- /dev/null +++ b/labs/converged-modeling-patterns/docker/init/01-grants.sql @@ -0,0 +1,15 @@ +-- Grant CMP_USER (created by the gvenzl APP_USER convention) the privileges the +-- patterns use. Runs as SYSDBA at initdb time (gvenzl convention). Idempotent. +ALTER SESSION SET CONTAINER = FREEPDB1; +BEGIN + BEGIN + EXECUTE IMMEDIATE 'CREATE USER cmp_user IDENTIFIED BY "CmpUser2026" QUOTA UNLIMITED ON users'; + EXCEPTION WHEN OTHERS THEN + IF SQLCODE != -1920 THEN RAISE; END IF; -- ORA-01920: user already exists (created by image) + END; + EXECUTE IMMEDIATE 'ALTER USER cmp_user QUOTA UNLIMITED ON users'; + EXECUTE IMMEDIATE 'GRANT CREATE SESSION, CREATE TABLE, CREATE VIEW, CREATE PROCEDURE, + CREATE TRIGGER, CREATE SEQUENCE, CREATE MATERIALIZED VIEW, CREATE PROPERTY GRAPH, + SODA_APP TO cmp_user'; +END; +/ diff --git a/labs/converged-modeling-patterns/docker/init/02-ords-enable.sql b/labs/converged-modeling-patterns/docker/init/02-ords-enable.sql new file mode 100644 index 0000000..51fca99 --- /dev/null +++ b/labs/converged-modeling-patterns/docker/init/02-ords-enable.sql @@ -0,0 +1,18 @@ +-- Documents the ORDS-enable of CMP_USER and covers the volume-reuse case where +-- ORDS is already installed. On FIRST boot this defers: gvenzl runs initdb scripts +-- BEFORE the entrypoint installs ORDS, so ORDS_METADATA does not exist yet — the +-- entrypoint re-runs the identical ENABLE_SCHEMA right after `ords install`. +ALTER SESSION SET CONTAINER = FREEPDB1; +BEGIN + EXECUTE IMMEDIATE q'[ + BEGIN + ORDS_METADATA.ORDS_ADMIN.ENABLE_SCHEMA(p_enabled => TRUE, p_schema => 'CMP_USER', + p_url_mapping_type => 'BASE_PATH', + p_url_mapping_pattern => 'cmp', p_auto_rest_auth => FALSE); + COMMIT; + END;]'; +EXCEPTION + WHEN OTHERS THEN + DBMS_OUTPUT.PUT_LINE('ORDS enable deferred (ORDS not installed yet): ' || SQLERRM); +END; +/ diff --git a/labs/converged-modeling-patterns/docker/scripts/entrypoint.sh b/labs/converged-modeling-patterns/docker/scripts/entrypoint.sh new file mode 100755 index 0000000..a75c693 --- /dev/null +++ b/labs/converged-modeling-patterns/docker/scripts/entrypoint.sh @@ -0,0 +1,101 @@ +#!/bin/bash +# Custom entrypoint: start Oracle (via the gvenzl base entrypoint), then install/ +# validate ORDS, ORDS-enable CMP_USER, turn on the Oracle API for MongoDB (TLS off +# for local dev), and serve ORDS. Both processes are monitored. +set -uo pipefail + +ORDS_CONFIG="/etc/ords/config" +ORDS_LOG="/tmp/ords.log" +ORACLE_PDB="FREEPDB1" +# Pool config lives in the container-local /etc/ords, not on the data volume, so a +# recreated container re-runs install (a no-op/validate against an existing schema). +ORDS_POOL_CONFIG="$ORDS_CONFIG/databases/default/pool.xml" + +# Start Oracle in the background using the base image entrypoint. +/opt/oracle/container-entrypoint.sh & +ORACLE_PID=$! + +echo "=== Waiting for Oracle to be ready ===" +until /opt/oracle/healthcheck.sh > /dev/null 2>&1; do sleep 2; done +echo "=== Oracle is ready ===" + +ADMIN_PWD="${ORACLE_PASSWORD:-Sandbox2026}" + +if [ ! -f "$ORDS_POOL_CONFIG" ]; then + echo "=== No ORDS pool config: installing/validating ORDS ===" + + # On a recreated container over an existing volume, the installer needs the + # ORDS_PUBLIC_USER password to rebuild the local wallet — reset it to a known + # value so the non-interactive install can proceed. + sqlplus -s / as sysdba <&1 || { + echo "=== ORDS install exited $?. Log tail: ===" + tail -50 /tmp/*.log 2>/dev/null || true + } +fi + +# ORDS-enable CMP_USER so its duality views + JSON collection tables surface as +# collections over the Mongo wire protocol. ENABLE_SCHEMA is an idempotent upsert. +echo "=== ORDS-enabling CMP_USER schema ===" +sqlplus -s / as sysdba <<'SQL' +ALTER SESSION SET CONTAINER = FREEPDB1; +BEGIN + ORDS_METADATA.ORDS_ADMIN.ENABLE_SCHEMA(p_enabled => TRUE, p_schema => 'CMP_USER', + p_url_mapping_type => 'BASE_PATH', + p_url_mapping_pattern => 'cmp', p_auto_rest_auth => FALSE); + COMMIT; +END; +/ +SQL + +# Turn the Mongo API on, plaintext (local dev). NB: the key is mongo.tls, not +# mongo.tls.enabled. Re-run every boot: /etc/ords is container-local. +ords --config "$ORDS_CONFIG" config set mongo.enabled true +ords --config "$ORDS_CONFIG" config set mongo.port 27017 +ords --config "$ORDS_CONFIG" config set mongo.tls false + +echo "=== Starting ORDS ===" +ords --config "$ORDS_CONFIG" serve --port 8181 > "$ORDS_LOG" 2>&1 & +ORDS_PID=$! + +echo "=== Waiting for ORDS health ===" +for i in $(seq 1 90); do + if curl -sf -o /dev/null -w '%{http_code}' http://localhost:8181/ 2>/dev/null | grep -q '302\|200'; then + echo "=== ORDS is healthy ==="; break + fi + [ "$i" -eq 90 ] && { echo "=== ORDS not healthy after 3 min. Log tail: ==="; tail -30 "$ORDS_LOG" 2>/dev/null || true; } + sleep 2 +done + +echo "=== Oracle + ORDS running. Monitoring... ===" +while true; do + if ! kill -0 "$ORACLE_PID" 2>/dev/null; then + echo "=== Oracle process died ==="; kill "$ORDS_PID" 2>/dev/null || true; exit 1 + fi + if ! kill -0 "$ORDS_PID" 2>/dev/null; then + echo "=== ORDS died, restarting ===" + ords --config "$ORDS_CONFIG" serve --port 8181 > "$ORDS_LOG" 2>&1 & + ORDS_PID=$! + fi + sleep 5 +done diff --git a/labs/converged-modeling-patterns/docker/scripts/install-ords.sh b/labs/converged-modeling-patterns/docker/scripts/install-ords.sh new file mode 100755 index 0000000..8806277 --- /dev/null +++ b/labs/converged-modeling-patterns/docker/scripts/install-ords.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# Install the latest ORDS into the image during `docker build`. Runs as root. +set -euo pipefail + +ORDS_HOME="/opt/oracle/ords" +ORDS_CONFIG="/etc/ords/config" + +echo "=== Installing ORDS (latest) ===" + +# ORDS needs a JRE; Java 17 is the current LTS ORDS supports. +microdnf install -y java-17-openjdk-headless unzip curl && microdnf clean all + +mkdir -p "$ORDS_HOME" "$ORDS_CONFIG" +cd /tmp +curl -fsSL -o ords.zip "https://download.oracle.com/otn_software/java/ords/ords-latest.zip" +unzip -q ords.zip -d "$ORDS_HOME" +rm ords.zip + +ln -sf "$ORDS_HOME/bin/ords" /usr/local/bin/ords + +export ORDS_CONFIG +ords config set --global config.dir "$ORDS_CONFIG" + +echo "=== ORDS installed: $(ords --version 2>/dev/null || echo 'version unknown') ===" diff --git a/labs/converged-modeling-patterns/docs/instructor-runbook.md b/labs/converged-modeling-patterns/docs/instructor-runbook.md new file mode 100644 index 0000000..1ad70d7 --- /dev/null +++ b/labs/converged-modeling-patterns/docs/instructor-runbook.md @@ -0,0 +1,125 @@ +# Instructor runbook: hands-on console in event mode + +Solo mode (the default) needs none of this. It's one user, no sign-in, nothing to +prepare. This runbook is for running the console with a room full of attendees. + +## Before the session (15 minutes) + +1. Set the event's secrets and open the console (only the console) to the room: + + ```bash + export LAB_MODE=event \ + ADMIN_PASSWORD='' \ + LAB_ADMIN_PASSWORD='' \ + EVENT_CODE='' \ + CMP_UI_BIND=0.0.0.0 + ``` + + The console refuses to start in event mode while `LAB_ADMIN_PASSWORD` is still the repo + default. `CMP_UI_BIND=0.0.0.0` publishes port 3100 on every interface so attendees + can reach it; by default it binds to `127.0.0.1` only. Passwords must not contain `"`. + + **Leave the database ports (1522, 8182, 27018) on localhost**: do not set + `CMP_DB_BIND`. The console reaches the database over the compose network and never + needs them published. Opening them to the room would hand attendees the database + itself: LAB_ADMIN (which creates and drops users) and SYS, with credentials that are + in this repo. +2. Start: `docker compose up -d`, or `./lab.sh up` where there is no compose provider + (Podman, see the README's *Podman* section). Both read `.env`. First start builds images. +3. Wait until `curl -fsS http://localhost:3100/api/config` shows `"mode":"event"`. +4. Open `http://:3100/admin.html`, sign in with `ADMIN_PASSWORD`, and + **pre-warm** about as many workspaces as you expect attendees (each takes a few + seconds; attendees who arrive early simply get a fresh one; you can pre-warm more + than once if the room grows). +5. Put the console URL and the event code on a slide. Attendees sign in with their name, + email and that code; each gets their own database schema. + +## Capacity + +The SQL lane has been load-tested at 150 simultaneous attendees, prewarmed, all clicking +**Run** in the same burst (20% doing their own write, 80% running the identical read +query): **0** requests came back `429`/`5xx`, **120 of 150** reads were served from the +cache instead of hitting the database, the database never had more than **1** attendee +session active at once (the gate held), and latency was **p50 ≈ 1.1 s / p95 ≈ 1.8 s**. +That measurement is SQL only: the MongoDB lane goes through the same gate, but it has not +been load-tested at room scale, so treat a room full of MongoDB runs as unmeasured. +That is `app/test/load/load.mjs`, run with prewarm against a console already started in +event mode. To repeat it for your own room size, from `app/`: + +```bash +LAB_URL=http://localhost:3100 EVENT_CODE='' ADMIN_PASSWORD='' \ + USERS=150 npm run test:load +``` + +`USERS` defaults to 150; `WRITERS` (fraction that write instead of read) defaults to +`0.2`. It needs a direct line to the database too (`DB_HOST`/`DB_PORT`, default +`localhost:1522`) to sample how many attendee sessions are active at once. + +## During the session + +- **Do not run `./run.sh` during an event.** Its last stage runs the console's + integration tests against the same database (it skips them while the console is in + event mode, but its first stage still runs `docker compose up -d` or `./lab.sh up`). +- **Pause execution** (admin page) while you talk through a slide. While paused, every + new run *and every run already waiting in the queue* is refused with "paused by + instructor"; a run already executing finishes. Nothing resumes on its own: after you + un-pause, attendees press **Run** again. +- The queue card shows what is waiting and what is running right now, with a + **Cancel** next to each running request. Cancel interrupts the statement executing + now (it returns `ORA-01013`) and the rest of that request's statements do not run + (`LAB-CANCELLED`). +- An attendee who broke their own workspace can press **Reset this pattern** on their + page. From the attendee table you can **Reset all patterns** for one of them (it + reports an error if any pattern fails to build), or use **Reset every attendee** to + rebuild every pattern in every workspace. That runs in the background: the admin page + shows "N of M done" and lists any workspace that failed. It queues behind attendee + work, so it can take a while with a full room. +- The time limit (10 s by default, SQL and MongoDB) applies to each **Run** as a whole, + not to each statement: a run of several statements shares one 10 s budget, and once + it is spent the statement executing is stopped and the rest return `LAB-TIMEOUT`. + It is adjustable live from the admin page, 1–60 s, if a demo genuinely needs longer. + Resets have their own 120 s budget. +- Point attendees at the ⓘ next to any card, tab or knob; the Measure-it curve shows + where their result sits against a reference run. + +## After the session + +- **End event** cuts off every attendee workspace at once: each schema is locked (no + new connections, whatever it was doing) and queued for dropping, and returns + `{ dropped: 0, pending: N }`. A background sweep does the dropping: its first pass runs + within 30 s and it retries every 30 s; after a workspace has sat pending for 2 minutes + it kills that schema's remaining database sessions (the Oracle API for MongoDB holds + idle sessions open past client disconnect) and drops it on the retry. The admin page + (or `GET /api/admin/status`, field `pending`) shows the count still being dropped; it + falls to `0` within a few minutes. +- Then `docker compose down` (or `./lab.sh down`). Adding `-v` **deletes the database volume** (all lab data). Only do that when you mean to start from scratch. + +## Limits to know + +- One statement executes at a time (a 1-permit gate); each run (all its statements + together) times out after 10 s by default; each result is capped at 500 rows / 1 MB (`truncated: true` beyond + that); a request waits at most 30 s in the queue before "busy: the database is + serving others, try again" (`429`). +- Oracle AI Database 26ai Free: 2 CPU threads, 2 GB RAM, 12 GB of user data. Each + attendee schema is capped at a 50 MB quota (ample for the patterns' scaled-down + data); ~150 of them fit comfortably. +- **Measure it** runs each side twice, both rolled back: one unmeasured warm-up, then the + measured pass (V$MYSTAT deltas, in-memory undo off for that session). It is a + single-session measurement on 26ai Free, not a fixed constant: + an `INSERT` can vary by roughly ±76 redo bytes / ±1 block run to run, and a large JSON + append can step by a whole LOB chunk. Say "same order of magnitude," not exact bytes, + when presenting it live. If the card shows a `LAB-MEASURE` error instead of numbers, + the session-level setting the measurement depends on failed to apply. Rerun it; if it + keeps failing, note it and move on rather than debugging live. +- Patterns 02 and 06 are deliberately scaled up so their write cost is visible: + subscriber `S-001` (Computed) carries 1,000 CDR line items, and whale `A-900` + (Outlier) embeds 800 clients, with the measured statement an append. Measured ratios + on 26ai Free: + + | Pattern | Document side | Converged side | Ratio | + |---|---|---|---| + | 02 – Computed | ~52 KB redo / 19 blocks | ~1.6 KB / 11 blocks | ~33× | + | 06 – Outlier | ~20 KB / 15 blocks | ~1.0–1.1 KB / 7–8 blocks | ~20× | + + These are read-modify-write costs of a growing document versus a small appended row, + not a claim about how either engine stores JSON internally. diff --git a/labs/converged-modeling-patterns/lab.sh b/labs/converged-modeling-patterns/lab.sh new file mode 100755 index 0000000..9cf510d --- /dev/null +++ b/labs/converged-modeling-patterns/lab.sh @@ -0,0 +1,147 @@ +#!/usr/bin/env bash +# ============================================================================ +# Compose-free launcher: runs the same stack as compose.yml with the plain +# container CLI, so the lab works on Podman with no compose provider installed +# (and on a bare Docker engine too). +# +# ./lab.sh up [--build] # build images if missing (or always, with --build), start both services +# ./lab.sh down [-v] # remove the containers; -v also DELETES the database volume +# ./lab.sh status # containers, health, console URL +# ./lab.sh logs [oracle|lab-ui] [podman logs flags...] +# ./lab.sh test # the lab console's vitest suites (run.sh stage 3 uses this) +# +# Engine: podman if installed, else docker. Override with CONTAINER_ENGINE=docker. +# Settings come from .env (same keys as compose.yml); a variable already set in the +# shell wins over .env, as with compose. +# +# KEEP IN STEP WITH compose.yml. compose.yml is the source of truth; this script +# mirrors its images, container names, ports, environment and volume. It uses the +# same volume and network names compose would create, so the database survives a +# switch between the two (remove the containers with one before starting the other: +# the container names are fixed). +# ============================================================================ +set -uo pipefail +cd "$(dirname "$0")" + +ENGINE="${CONTAINER_ENGINE:-$(command -v podman >/dev/null && echo podman || echo docker)}" +PROJECT=converged-modeling-patterns +NET="${PROJECT}_default" +VOL="${PROJECT}_cmp-data" +DB=cmp-oracle +UI=cmp-lab-ui +IMG_DB="${PROJECT}-oracle:latest" +IMG_UI="${PROJECT}-lab-ui:latest" +IMG_TEST="${PROJECT}-lab-ui-test:latest" + +# .env: KEY=VALUE lines, no shell evaluation. The shell environment wins. +if [ -f .env ]; then + while IFS= read -r line || [ -n "$line" ]; do + line="${line%$'\r'}" + [[ "$line" =~ ^[[:space:]]*(#|$) ]] && continue + key="${line%%=*}"; val="${line#*=}" + [[ "$key" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || continue + val="${val#\"}"; val="${val%\"}"; val="${val#\'}"; val="${val%\'}" + [ -z "${!key+x}" ] && export "$key=$val" + done < .env +fi + +# compose.yml defaults +: "${CMP_DB_BIND:=127.0.0.1}" "${CMP_PORT:=1522}" "${CMP_ORDS_PORT:=8182}" "${CMP_MONGO_PORT:=27018}" +: "${CMP_UI_BIND:=127.0.0.1}" "${CMP_UI_PORT:=3100}" +: "${ORACLE_PASSWORD:=Sandbox2026}" "${CMP_PASSWORD:=CmpUser2026}" "${LAB_ADMIN_PASSWORD:=LabAdmin2026}" +: "${LAB_MODE:=solo}" "${ADMIN_PASSWORD:=}" "${EVENT_CODE:=}" "${DB_POOL_MAX:=1}" "${MONGO_POOL_MAX:=1}" + +e() { "$ENGINE" "$@"; } +exists() { e container inspect "$1" >/dev/null 2>&1; } +running() { [ "$(e container inspect -f '{{.State.Running}}' "$1" 2>/dev/null)" = true ]; } +has_img() { e image inspect "$1" >/dev/null 2>&1; } + +build() { # $1 = force (1/0) + if [ "$1" = 1 ] || ! has_img "$IMG_DB"; then + echo "==> Building $IMG_DB (first build pulls the 26ai Free base image: several minutes) ..." + e build -t "$IMG_DB" -f docker/Dockerfile docker || exit 1 + fi + if [ "$1" = 1 ] || ! has_img "$IMG_UI"; then + echo "==> Building $IMG_UI ..." + e build -t "$IMG_UI" -f app/Dockerfile --target runtime . || exit 1 + fi +} + +ensure_net_vol() { + # compose's own labels, so a later `compose up` adopts these instead of warning + local p=(--label "com.docker.compose.project=$PROJECT") + e network inspect "$NET" >/dev/null 2>&1 || e network create "${p[@]}" --label com.docker.compose.network=default "$NET" >/dev/null || exit 1 + e volume inspect "$VOL" >/dev/null 2>&1 || e volume create "${p[@]}" --label com.docker.compose.volume=cmp-data "$VOL" >/dev/null || exit 1 +} + +start_db() { + if running "$DB"; then return; fi + if exists "$DB"; then echo "==> Starting $DB ..."; e start "$DB" >/dev/null || exit 1; return; fi + echo "==> Creating $DB ..." + e run -d --name "$DB" --network "$NET" --network-alias oracle \ + -p "$CMP_DB_BIND:$CMP_PORT:1521" -p "$CMP_DB_BIND:$CMP_ORDS_PORT:8181" -p "$CMP_DB_BIND:$CMP_MONGO_PORT:27017" \ + -e ORACLE_PASSWORD="$ORACLE_PASSWORD" -e APP_USER=CMP_USER -e APP_USER_PASSWORD="$CMP_PASSWORD" \ + --health-cmd healthcheck.sh --health-interval 10s --health-timeout 10s --health-retries 60 --health-start-period 180s \ + -v "$VOL:/opt/oracle/oradata" \ + "$IMG_DB" >/dev/null || exit 1 +} + +wait_db() { # compose's depends_on: service_healthy, done by polling the same check + echo -n "==> Waiting for the database to be healthy " + for _ in $(seq 1 120); do + e exec "$DB" healthcheck.sh >/dev/null 2>&1 && { echo " ready."; return; } + running "$DB" || { echo " $DB stopped. See: ./lab.sh logs oracle"; exit 1; } + echo -n '.'; sleep 5 + done + echo " timed out after 10 minutes. See: ./lab.sh logs oracle"; exit 1 +} + +ui_env=(LAB_MODE ADMIN_PASSWORD EVENT_CODE ORACLE_PASSWORD CMP_PASSWORD LAB_ADMIN_PASSWORD DB_POOL_MAX MONGO_POOL_MAX) +ui_config() { # fingerprint of everything that should force a console recreate (compose recreates on change) + { for k in "${ui_env[@]}"; do printf '%s=%s\n' "$k" "${!k}"; done + echo "$CMP_UI_BIND:$CMP_UI_PORT"; e image inspect -f '{{.Id}}' "$IMG_UI"; } | cksum | cut -d' ' -f1 +} + +start_ui() { + local want; want="$(ui_config)" + if exists "$UI"; then + if [ "$(e container inspect -f '{{index .Config.Labels "cmp.config"}}' "$UI" 2>/dev/null)" = "$want" ]; then + running "$UI" || { echo "==> Starting $UI ..."; e start "$UI" >/dev/null || exit 1; } + return + fi + echo "==> Settings changed: recreating $UI ..."; e rm -f "$UI" >/dev/null + else + echo "==> Creating $UI ..." + fi + local args=(); for k in "${ui_env[@]}"; do args+=(-e "$k=${!k}"); done + e run -d --name "$UI" --network "$NET" --label "cmp.config=$want" \ + -p "$CMP_UI_BIND:$CMP_UI_PORT:3000" "${args[@]}" "$IMG_UI" >/dev/null || exit 1 +} + +status() { + e ps -a --filter "name=^($DB|$UI)\$" --format '{{.Names}}\t{{.Status}}\t{{.Ports}}' + running "$UI" && echo "Console: http://localhost:$CMP_UI_PORT Admin: http://localhost:$CMP_UI_PORT/admin.html (mode: $LAB_MODE)" +} + +case "${1:-}" in + up) + [ "$LAB_MODE" = event ] && [ -z "$ADMIN_PASSWORD" ] && { echo "LAB_MODE=event needs ADMIN_PASSWORD (set it in .env)."; exit 1; } + build "$([ "${2:-}" = --build ] && echo 1 || echo 0)" + ensure_net_vol; start_db; wait_db; start_ui; status ;; + down) + e rm -f "$UI" "$DB" >/dev/null 2>&1 + if [ "${2:-}" = -v ]; then e volume rm "$VOL" >/dev/null 2>&1 && echo "Removed containers and the database volume." + else echo "Removed containers (database volume kept; add -v to delete it)."; fi ;; + status) status ;; + logs) + svc="${2:-oracle}"; shift $(( $# >= 2 ? 2 : 1 )) + case "$svc" in oracle) e logs "$@" "$DB" ;; lab-ui) e logs "$@" "$UI" ;; *) echo "logs: oracle | lab-ui"; exit 1 ;; esac ;; + test) + running "$DB" || { echo "$DB is not running: ./lab.sh up first."; exit 1; } + e build -t "$IMG_TEST" -f app/Dockerfile --target test . || exit 1 + e run --rm --network "$NET" \ + -e DB_HOST=oracle -e DB_PORT=1521 -e MONGO_HOST=oracle -e MONGO_PORT=27017 \ + -e ORACLE_PASSWORD="$ORACLE_PASSWORD" -e CMP_PASSWORD="$CMP_PASSWORD" -e LAB_ADMIN_PASSWORD="$LAB_ADMIN_PASSWORD" \ + "$IMG_TEST" ;; + *) sed -n '2,13p' "$0" | sed 's/^# \{0,1\}//'; exit 1 ;; +esac diff --git a/labs/converged-modeling-patterns/patterns/01-extended-reference/01-document-model.sql b/labs/converged-modeling-patterns/patterns/01-extended-reference/01-document-model.sql new file mode 100644 index 0000000..f45fa2a --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/01-extended-reference/01-document-model.sql @@ -0,0 +1,86 @@ +-- ============================================================================ +-- Pattern 01 · Extended Reference · THE DOCUMENT MODEL (the starting point) +-- Industry: wealth management. Access pattern: open any client or account and +-- show the servicing advisor's name, office, and desk line. No join needed. +-- +-- The document move: copy the most-read advisor fields INTO every client doc so +-- the read never has to look them up. "Only copy fields that rarely change." +-- +-- On a read-heavy workload this is a correct bet: about 50M client-360 reads a +-- day against ~300 advisor edits, and four consumers want the card inline. +-- +-- THE WRITE-AMPLIFICATION COST (made explicit below): the advisor is now stored +-- once per document that embeds it. The moment a copied value changes (advisor +-- moves offices, changes desk line) you must find and rewrite EVERY embedded copy. +-- In the deck's book that is ~2,700 documents per advisor (790 client docs plus +-- 1,920 account docs), not one transaction, and until the last rewrite lands the +-- client portal and the AI copilot can read different offices. +-- "Write amplification in, update anomaly out." +-- +-- Runs SQL-native (JSON Collection Table) on the same 26ai engine, so the +-- comparison is about shape, not vendor; the same shape is what a MongoDB-API +-- developer would model as a collection. +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @01-document-model.sql +-- ============================================================================ + +-- Idempotent: drop prior run cleanly. +BEGIN + FOR r IN (SELECT table_name FROM user_tables WHERE table_name = 'XR_CLIENT_DOC') LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' PURGE'; + END LOOP; +END; +/ + +CREATE JSON COLLECTION TABLE xr_client_doc; + +-- Each client doc carries an EMBEDDED COPY of the advisor block (extended +-- reference). Two clients of advisor A-001 => two copies of A-001's office/desk. +INSERT INTO xr_client_doc VALUES (JSON('{"_id":"C-001","fullName":"Ada Client","segment":"HNW", + "advisor":{"advisorId":"A-001","fullName":"Grace Advisor","office":"NYC-01","desk":"+1-212-555-0101"}}')); +INSERT INTO xr_client_doc VALUES (JSON('{"_id":"C-002","fullName":"Ben Client","segment":"MASS", + "advisor":{"advisorId":"A-001","fullName":"Grace Advisor","office":"NYC-01","desk":"+1-212-555-0101"}}')); +INSERT INTO xr_client_doc VALUES (JSON('{"_id":"C-003","fullName":"Cara Client","segment":"HNW", + "advisor":{"advisorId":"A-002","fullName":"Alan Advisor","office":"CHI-04","desk":"+1-312-555-0144"}}')); +COMMIT; + +-- The read the pattern optimizes for: zero-join, everything inline. This is why +-- people reach for the embed. +-- @step Read the embedded advisor block +-- @note Zero-join read: every client doc already carries its own copy of the advisor's office and desk. +-- @why Read/write is why teams embed: at 50 million client-360 reads a day against about 300 advisor edits, a read with no lookup is the right thing to optimize. +-- @look Every client row returns its own advisor_office, and the statement touches one table. +-- @figure doc-shape.svg The client document as built: the advisor card copied into every client and account +SELECT JSON_VALUE(data, '$.fullName') AS client, + JSON_VALUE(data, '$.advisor.office') AS advisor_office +FROM xr_client_doc; + +-- --------------------------------------------------------------------------- +-- THE NEEDLE FLIPS HERE. Advisor A-001 moves from NYC-01 to NYC-09. There is no +-- advisor record to update: the advisor only exists as copies. So the write +-- fans out across EVERY client doc that embeds A-001, and each JSON_TRANSFORM +-- re-serializes the whole document into a new block (WiredTiger copy-on-write; +-- OSON re-encode on Oracle). 2 rows here; ~2,700 documents per advisor on the +-- deck's book, and ~22M rewrites on a reorg day of 8,000 changes. +-- --------------------------------------------------------------------------- +-- @step Move advisor A-001 to a new office +-- @note Every client document that embeds A-001 gets rewritten: one logical change fans out to every embedded copy, not to one row. +-- @why Update locality is the hot knob: the advisor exists only as copies, so one office change is a read-modify-write of every document that embeds A-001. At 2,700 embedding documents per advisor, that is 2,700 rewrites per edit. +-- @look Rows affected is 2 here, one per embedded copy; Measure it shows the redo those copies cost. +-- @measure advisor-move +UPDATE xr_client_doc c +SET data = JSON_TRANSFORM(data, + SET '$.advisor.office' = 'NYC-09', + SET '$.advisor.desk' = '+1-212-555-0999') +WHERE JSON_VALUE(data, '$.advisor.advisorId') = 'A-001'; +-- ^ rows updated = every client of A-001. Miss one (or crash mid-fan-out) and +-- that client doc now serves a STALE office to the portal and to the copilot. +COMMIT; + +-- @step Confirm the fan-out +-- @note Count of client docs now showing the new office: one logical change touched every embedded copy. +-- @why Update locality again: the count is how far one logical change had to travel. At 2,700 copies per advisor that fan-out is a job, not one transaction, and until it finishes the portal and the copilot can read different offices. +-- @look The write_amplification line reports 2 client docs, the number that becomes 2,700 at production scale. +SELECT 'document model: advisor change fanned out to ' || + COUNT(*) || ' client docs' AS write_amplification +FROM xr_client_doc +WHERE JSON_VALUE(data, '$.advisor.office') = 'NYC-09'; diff --git a/labs/converged-modeling-patterns/patterns/01-extended-reference/02-converged.sql b/labs/converged-modeling-patterns/patterns/01-extended-reference/02-converged.sql new file mode 100644 index 0000000..900c0b8 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/01-extended-reference/02-converged.sql @@ -0,0 +1,105 @@ +-- ============================================================================ +-- Pattern 01 · Extended Reference · THE CONVERGED ALTERNATIVE +-- "Project, don't copy." +-- +-- The advisor is stored ONCE, in its own table. Each client references it by FK. +-- A JSON Relational Duality View projects the SAME client document the document +-- developer wanted (advisor block inline), but the advisor block is assembled +-- through the FK at read time, not stored as a copy. +-- +-- THE TRADEOFF: every read pays a primary-key join (about 15% more read work in +-- the deck's model); in exchange an advisor moving offices is ONE row, and every +-- client document that projects that advisor reflects it at the same commit, with +-- no fan-out and no stale copy. The read shape is identical; the write cost goes +-- from O(documents embedding the advisor) to O(1). On a normal day (~300 edits) +-- embedding is still cheaper; break-even is ~930 edits a day, and a reorg day +-- (8,000) is where projecting wins on cost and on correctness. +-- +-- When the document shape stays: snapshot the IMMUTABLE (a trade's execution +-- price, the advisor of record on a signed statement: freeze it in the doc), +-- project the MUTABLE (an advisor's CURRENT office is reference data: project it). +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @02-converged.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables + WHERE table_name IN ('XR_CLIENTS','XR_ADVISORS') + ORDER BY table_name) LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' CASCADE CONSTRAINTS PURGE'; + END LOOP; +END; +/ + +-- Relational projection: the advisor is stored ONCE, as one row. +CREATE TABLE xr_advisors ( + advisor_id VARCHAR2(12) PRIMARY KEY, + full_name VARCHAR2(120) NOT NULL, + office VARCHAR2(24) NOT NULL, + desk VARCHAR2(24) NOT NULL +); + +CREATE TABLE xr_clients ( + client_id VARCHAR2(12) PRIMARY KEY, + full_name VARCHAR2(120) NOT NULL, + segment VARCHAR2(12) NOT NULL, + primary_advisor_id VARCHAR2(12) NOT NULL REFERENCES xr_advisors(advisor_id) +); + +INSERT INTO xr_advisors VALUES ('A-001','Grace Advisor','NYC-01','+1-212-555-0101'); +INSERT INTO xr_advisors VALUES ('A-002','Alan Advisor','CHI-04','+1-312-555-0144'); +INSERT INTO xr_clients VALUES ('C-001','Ada Client','HNW','A-001'); +INSERT INTO xr_clients VALUES ('C-002','Ben Client','MASS','A-001'); +INSERT INTO xr_clients VALUES ('C-003','Cara Client','HNW','A-002'); +COMMIT; + +-- The document the document dev wanted: assembled at read time from the rows. +-- Advisor block is WITH NOUPDATE: it is a PROJECTION of reference data, not a +-- copy this view is allowed to overwrite (per-field governance, engine-enforced: +-- writing advisor.office through the view fails with ORA-40940). +CREATE OR REPLACE JSON RELATIONAL DUALITY VIEW xr_client_dv AS +SELECT JSON { + '_id' : c.client_id, + 'fullName' : c.full_name WITH UPDATE, + 'segment' : c.segment WITH UPDATE, + 'advisor' : ( SELECT JSON { 'advisorId' : a.advisor_id, + 'fullName' : a.full_name, + 'office' : a.office, + 'desk' : a.desk } + FROM xr_advisors a WITH NOUPDATE + WHERE a.advisor_id = c.primary_advisor_id ) +} FROM xr_clients c WITH INSERT UPDATE DELETE; + +-- Same read the document model served: identical shape, zero stored copies. +-- @step Read the projected client document +-- @note Same shape as the document model's read: the advisor block is a live projection through the FK, not a stored copy. +-- @why Diversity is served without copies: the duality view assembles the advisor block through the foreign key on every read. That primary-key join is real read work, about 15% more per read in the deck's model. +-- @look The same client and advisor_office columns as the document model's read, now from xr_client_dv. +-- @figure erd.svg Advisor, client, account and trade in canonical form, each fact once, beside the fan-out one advisor change causes in the embedded model +SELECT JSON_VALUE(data, '$.fullName') AS client, + JSON_VALUE(data, '$.advisor.office') AS advisor_office +FROM xr_client_dv; + +-- --------------------------------------------------------------------------- +-- THE NEEDLE-FLIP, RESOLVED. Advisor A-001 moves offices: ONE row. Every client +-- document that projects A-001 is correct at the same commit. No fan-out, no +-- stale copy, no CDC job to reconcile the portal with the copilot. The price was +-- paid on the read side, as a join on every document read. +-- --------------------------------------------------------------------------- +-- @step Move advisor A-001 to a new office +-- @note One row: every client document that projects A-001 is correct in the same transaction, no fan-out, no stale copy. +-- @why Update locality collapses: the advisor is one row, so an office change is one small row write however many clients project it. +-- @look Rows affected is 1; compare its redo with the document model's in Measure it. +-- @measure advisor-move +UPDATE xr_advisors SET office = 'NYC-09', desk = '+1-212-555-0999' +WHERE advisor_id = 'A-001'; +COMMIT; + +-- @step Confirm every client reflects the move +-- @note Both C-001 and C-002 now read NYC-09: from one write to one row. +-- @why Update locality, resolved: both clients project the advisor row live, so they changed at the same commit with nothing to reconcile. +-- @look C-001 and C-002 both show NYC-09 in advisor_office_now. +SELECT JSON_VALUE(data, '$._id') AS client, + JSON_VALUE(data, '$.advisor.office') AS advisor_office_now +FROM xr_client_dv +ORDER BY 1; +-- ^ C-001 and C-002 both read NYC-09: from one write to one row. diff --git a/labs/converged-modeling-patterns/patterns/01-extended-reference/03-parity.js b/labs/converged-modeling-patterns/patterns/01-extended-reference/03-parity.js new file mode 100644 index 0000000..88484de --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/01-extended-reference/03-parity.js @@ -0,0 +1,44 @@ +// ============================================================================ +// Pattern 01 · Extended Reference · CROSS-API PARITY (mongosh): the flagship proof +// One projection, two access surfaces, one projected shape. The SAME client +// document from the duality view, read TWO ways: +// * SQL lane: SELECT ... FROM xr_client_dv (injected as $SQL_RESULT by run.sh) +// * Mongo lane: db.xr_client_dv.findOne({_id:"C-001"}) over the Mongo wire +// Asserts the documents are byte-equal after canonicalizing JSON and ignoring the +// duality _metadata (etag/asof). Exits non-zero on mismatch. +// ============================================================================ +function canon(x) { + if (x === null || x === undefined) return null; + if (Array.isArray(x)) return x.map(canon).sort((a, b) => + JSON.stringify(a) < JSON.stringify(b) ? -1 : 1); + if (typeof x === "object") { + const cn = x.constructor && x.constructor.name; + if (x._bsontype || /Decimal128|Double|Long|Int32/.test(cn || "")) return Number(x.toString()); + const o = {}; + for (const k of Object.keys(x).sort()) { if (k === "_metadata") continue; o[k] = canon(x[k]); } + return o; + } + return x; +} +const eq = (a, b) => JSON.stringify(canon(a)) === JSON.stringify(canon(b)); + +const viaSql = JSON.parse(process.env.SQL_RESULT || "null"); +if (!viaSql) { print("[FAIL] SQL_RESULT not provided by run.sh"); quit(1); } + +// @step Read the projected document over the MongoDB API +// @note Same client document the SQL lane read: the advisor block projects live through the duality view, whichever wire protocol you read it over. +// @why Diversity without copies: the portal keeps its driver and find(), the copilot reads SQL/JSON, and both read the same advisor row through one duality view. +// @look The returned document's advisor.office matches the SQL read, with a _metadata etag the duality view adds. +// @figure flow.svg One duality view serves the portal over the MongoDB API, the copilot over SQL/JSON and statements over SQL +const viaMongo = db.xr_client_dv.findOne({ _id: "C-001" }); +if (!viaMongo) { print("[FAIL] MongoDB API returned no document for C-001 (is xr_client_dv exposed?)"); quit(1); } + +if (!eq(viaSql, viaMongo)) { + print("[FAIL] cross-API document mismatch for client C-001"); + print(" SQL lane: " + JSON.stringify(canon(viaSql))); + print(" Mongo lane: " + JSON.stringify(canon(viaMongo))); + quit(1); +} +print("[PASS] xr_client_dv document byte-equal across SQL and MongoDB API (C-001)"); +print(" advisor.office via Mongo lane = " + viaMongo.advisor.office + " (projected, not copied)"); +quit(0); diff --git a/labs/converged-modeling-patterns/patterns/01-extended-reference/README.md b/labs/converged-modeling-patterns/patterns/01-extended-reference/README.md new file mode 100644 index 0000000..0e93813 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/01-extended-reference/README.md @@ -0,0 +1,128 @@ +--- +title: Extended Reference +industry: Wealth management +deck: "14–17" +problem: >- + Four consumers render every client with their advisor, so the advisor card is embedded + in every client and account document. Reads are instant, until an advisor moves offices + and every embedded copy has to be found and rewritten. +knobs: + - { name: Diversity, setting: "High: 4 consumers", help: "The client portal, the advisor CRM, the statement run and the AI copilot all render the client with the advisor card, so four readers want it inline." } + - { name: Read / write, setting: "Extreme read skew", help: "About 50 million client-360 reads a day against roughly 300 advisor edits: on this knob alone, embedding the card is the right bet." } + - { name: Update locality, setting: "Rare, fans out ×2,700", hot: true, help: "Each advisor edit lands on about 2,700 documents (790 clients plus 1,920 accounts), so a reorg day of 8,000 changes is roughly 22 million document rewrites." } +help: + tabs: + document: + why: "The starting point: copy the advisor card into every client document so the client-360 read needs no lookup. The bet pays until an advisor moves and every copy has to be found and rewritten." + look: "The advisor move's rows affected: one per client document that embeds A-001." + converged: + why: "The advisor is stored once and a duality view projects the same client document through the foreign key. Every read pays for a primary-key join (about 15% more read work in the deck's model); every advisor move is one row." + look: "The advisor move reports 1 row affected, and the confirm query still shows both of A-001's clients on NYC-09." + mongo: + why: "The same duality view, read over the MongoDB API: one set of rows, two access surfaces, one advisor row." + look: "The advisor.office in the returned document matches what the SQL read of xr_client_dv shows." + measure: + why: "The ratio is the document model's redo for the advisor move divided by the converged model's. The document side rewrites every document that embeds the advisor, so the ratio grows with that count; the converged side is one row at any count." + look: "Your dot at 2 embedded copies, on a reference line that reaches about 3,000× at 1,000 copies." +measure: + x_label: "documents embedding the moved advisor" + lab_x: 2 + deck_slides: "14–17" + calibration: + - { x: 1, ratio: 3.47 } # doc 1652 B, conv 476 B + - { x: 10, ratio: 30.71 } # doc 14616 B, conv 476 B + - { x: 100, ratio: 303.83 } # doc 144624 B, conv 476 B + - { x: 1000, ratio: 3036.59 } # doc 1445416 B, conv 476 B +--- +# Pattern 01: Extended Reference ⭐ + +**Wealth management.** Open any client or account and show the servicing advisor's +name, office, and desk line. The deck's book: 2,400 advisors, 1.9 million clients, +4.6 million accounts, and four consumers (client portal, advisor CRM, nightly +statement run, AI copilot) that all render the client with the advisor card. + +> Project, don't copy. + +## The document bet + +The Extended Reference pattern copies the most-read fields of a referenced entity +*into* the referencing document, so the read never has to look them up. The advice +is disciplined: "only copy fields that rarely change." Embed the advisor card on +every client and every account (statements render per account), and each screen is +a single document read with no lookup. + +On two of the three knobs this is the right bet. **Diversity** is high: four +consumers want the card inline. **Read / write** is extreme: about 50 million +client-360 reads a day (600 a second on average, 12,000 at the open) against roughly +300 advisor edits. Nothing about the choice is a mistake. + +## Where the needle flips + +The third knob, **update locality**, is where it turns. The advisor is now stored +once per document that embeds it, and nothing enforces that the copies agree. When a +copied value changes (an advisor moves offices, changes a desk line) every embedded +copy has to be found and rewritten. In the deck's book each advisor appears in about +**2,700 documents** (790 client documents plus 1,920 account documents), so one +office move is about 2,700 rewrites, each one re-serializing the whole document. +Twice a year a regional reorg makes about 8,000 such changes in a day: roughly +**22 million document rewrites**. + +Those rewrites are not one transaction. Until the last one lands, the portal can show +the old office while the copilot, reading another document as context, shows the new +one. Nothing crashes; the answers just disagree. + +That is the pattern's trade in one line: **write amplification in, update anomaly +out.** The illustrative cost model on deck slide 16 puts a number on it: embedding +costs one document read per screen plus 2,700 rewrites per change; projecting the +card costs about **15% more per read** plus one row per change. On a normal day (300 +changes) **embedding is cheaper**. Break-even is about **930 changes a day**. On reorg +day the day's total work roughly doubles, and correctness is the bigger cost. The +needle flips on the worst day, not the average one, so model for the reorg. + +Both sides of that curve run on Oracle (the document side here is a JSON collection +table on the same engine), so the comparison is about shape, not vendor. + +## The converged alternative + +Store the advisor **once**, as one row in the relational projection, and have each +client reference it by foreign key. A **JSON Relational Duality View** then projects +the exact client document the document developer wanted, advisor block inline. The +advisor block is assembled through the foreign key when the document is read, not +stored as a copy. + +What that costs: every read pays for a primary-key join, about 15% more read work in +the deck's model. Joins are never free. What it buys: an advisor moving offices is +**one row**, and every document that projects that advisor shows the change at the +same commit. No fan-out, no stale copy, no sync job between the portal and the +copilot. + +The view also decides what a write through it can touch. `fullName` and `segment` are +`WITH UPDATE`; the advisor subquery is `WITH NOUPDATE`, so an attempt to change +`advisor.office` through the view is rejected (ORA-40940) and the office changes only +in its own row. That is per-field governance enforced by the engine. + +## When the document shape stays + +- **Snapshot the immutable.** A trade's execution price, or the advisor of record on + a signed statement, is history: freeze it in the document. A later change must + never rewrite it. That is Extended Reference used exactly as intended. +- **Project the mutable.** An advisor's *current* office is reference data that + changes: project it rather than copy it. +- If edits stay well below the break-even and there is no reorg-style burst, the + embedded card is cheaper to read and the extra rewrites are a fair price. + +## In this folder + +| File | What it shows | +|---|---| +| `01-document-model.sql` | Advisor card embedded in every client document; an office change fans out across every copy | +| `02-converged.sql` | Advisor stored once as a row; a duality view projects it into the client document; the office change is one row | +| `03-parity.js` ⭐ | Reads `xr_client_dv` through the **MongoDB API** and asserts it is byte-equal to the same view read through **SQL**: one projection, two access surfaces, the same projected shape | +| `_capture.sql` | Helper: emits the SQL-lane document that `03-parity.js` compares against | + +## Validated + +Both lanes run clean on **Oracle AI Database 26ai Free (`23.26.3-faststart`)**. The +cross-API parity assertion passes: the projected client document is identical through +SQL and through the Oracle API for MongoDB. +Run with `../../run.sh 01-extended-reference`. diff --git a/labs/converged-modeling-patterns/patterns/01-extended-reference/_capture.sql b/labs/converged-modeling-patterns/patterns/01-extended-reference/_capture.sql new file mode 100644 index 0000000..2e2031f --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/01-extended-reference/_capture.sql @@ -0,0 +1,9 @@ +-- Capture the projected client document AS SEEN FROM SQL (the duality view +-- xr_client_dv), serialized to one JSON string. run.sh injects this into +-- 03-parity.js as $SQL_RESULT to compare against the same document read through +-- the MongoDB API. +SET HEADING OFF PAGESIZE 0 FEEDBACK OFF VERIFY OFF TRIMSPOOL ON +SET LINESIZE 32767 LONG 100000000 LONGCHUNKSIZE 1000000 +SELECT JSON_SERIALIZE(data RETURNING CLOB) +FROM xr_client_dv +WHERE JSON_VALUE(data, '$._id') = 'C-001'; diff --git a/labs/converged-modeling-patterns/patterns/01-extended-reference/calibrate.sql b/labs/converged-modeling-patterns/patterns/01-extended-reference/calibrate.sql new file mode 100644 index 0000000..94f30f3 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/01-extended-reference/calibrate.sql @@ -0,0 +1,14 @@ +-- Calibration resize for Measure it: advisor A-001 serves :n clients on BOTH sides. +-- Document model: :n client docs embed a copy of A-001. Converged: :n client rows +-- reference the one A-001 row. C-003 (advisor A-002) stays as in the lab. +DELETE FROM xr_client_doc WHERE JSON_VALUE(data, '$.advisor.advisorId') = 'A-001'; +INSERT INTO xr_client_doc (data) +SELECT JSON_OBJECT('_id' VALUE 'C-' || TO_CHAR(100000 + LEVEL), 'fullName' VALUE 'Client ' || LEVEL, 'segment' VALUE 'HNW', + 'advisor' VALUE JSON_OBJECT('advisorId' VALUE 'A-001', 'fullName' VALUE 'Grace Advisor', + 'office' VALUE 'NYC-01', 'desk' VALUE '+1-212-555-0101') RETURNING JSON) +FROM dual CONNECT BY LEVEL <= :n; +DELETE FROM xr_clients WHERE primary_advisor_id = 'A-001'; +INSERT INTO xr_clients (client_id, full_name, segment, primary_advisor_id) +SELECT 'C-' || TO_CHAR(100000 + LEVEL), 'Client ' || LEVEL, 'HNW', 'A-001' +FROM dual CONNECT BY LEVEL <= :n; +COMMIT; diff --git a/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/doc-shape.svg b/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/doc-shape.svg new file mode 100644 index 0000000..e34f940 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/doc-shape.svg @@ -0,0 +1 @@ +clients · one document per client{"_id": "C-104233","name": "Ada Okafor","segment": "HNW","advisor": {"id": "A-317","name": "Grace Hopper","office": "Leeds","desk": "+44 113 555 0101"},"accounts": [{ "id": "ACC-88", "type": "ISA","advisor": { "id": "A-317","office": "Leeds" } },{ "id": "ACC-91", "type": "GIA", ... }]}advisor card, copied~790 client docscarry this copycopied again peraccount: ~1,920more copies \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/erd.svg b/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/erd.svg new file mode 100644 index 0000000..174329e --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/erd.svg @@ -0,0 +1 @@ +CANONICAL FORM: EACH FACT ONCETHE SAME FACT, EMBEDDEDADVISORPKadvisor_idvarchar2full_namevarchar2officevarchar2desk_phonevarchar2regionvarchar2CLIENTPKclient_idvarchar2full_namevarchar2segmentvarchar2FKadvisor_idvarchar2emailvarchar2ACCOUNTPKaccount_idvarchar2FKclient_idvarchar2acct_typevarchar2balancenumberTRADEPKtrade_idvarchar2FKaccount_idvarchar2symbolvarchar2quantitynumberexec_pricenumberadvisors · A-317office: Leeds → Manchester790client docs1,920account docs≈ 2,700 document rewritesper change · not one transactionRelational projection1 row update · atomicevery consumer sees itmutable reference data: copying it is what creates write amplificationimmutable snapshot: safe to copy \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/flow.svg b/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/flow.svg new file mode 100644 index 0000000..9ba13f7 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/flow.svg @@ -0,0 +1 @@ +RELATIONAL PROJECTIONPROJECTIONCONSUMERSadvisorsoffice lives hereclientsadvisor_id FKaccountsclient_id FKclient_dvduality viewadvisor @noupdatePortalMongo API find()AI copilotSQL/JSONStatementsSQL batch \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/model-curve.svg b/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/model-curve.svg new file mode 100644 index 0000000..fc5e720 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/01-extended-reference/figures/model-curve.svg @@ -0,0 +1 @@ +KNOB SETTINGS · THIS WORKLOAD1 · DiversityHigh: 4 consumersportal, CRM, statements andcopilot all read the card2 · Read / writeExtreme read skew5×10⁷ reads/day against~300 advisor edits/day3 · Update localityRare, fans out ×2,700each edit lands on everydocument that copied itDAILY WORK vs ADVISOR CHANGES / DAY · illustrative modelembedding is cheaperprojection is cheaper50M100M200M500M1B101001k10k100kadvisor changes per day (log)work units per day (log)break-even ≈ 930 / daynormal day · 300reorg day · 8,000Embed the cardProject it · duality view, +15% read \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/02-computed/01-document-model.sql b/labs/converged-modeling-patterns/patterns/02-computed/01-document-model.sql new file mode 100644 index 0000000..0ba999a --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/02-computed/01-document-model.sql @@ -0,0 +1,111 @@ +-- ============================================================================ +-- Pattern 02 · Computed · THE DOCUMENT MODEL (the starting point) +-- Industry: telecom. Access pattern: show a subscriber's current-cycle usage +-- (total MB, minutes, cost) instantly on the account page, and rank the top +-- talkers for the ops dashboard. +-- +-- The document move: bake the rollup onto the subscriber doc. Compute once on +-- write, then serve every read from the stored answer instead of re-aggregating. +-- The pattern's own math: 1M reads/hr vs 1K writes/hr => "compute on write +-- divides work by 1000." The read side of this bet is right. +-- +-- THE WRITE-AMPLIFICATION COST: usage is not written once. Every Call Detail +-- Record (CDR) appends a line item to the subscriber's usage[] and re-ticks the +-- rollup, and each one is a read-modify-write of the WHOLE subscriber document. +-- The document grows with every call, so the redo per CDR grows with it: here +-- S-001 already carries 1,000 CDR line items this cycle. The pattern's escape +-- hatch for a hot subscriber, counter sharding (split the counter into N shard +-- docs), spreads the write contention, but every balance check then reads N +-- docs and sums them: a write problem traded for a read problem. And the Top-N +-- dashboard has no rollup to lean on across subscribers: it must +-- SUM -> SORT -> LIMIT over the collection every load. +-- +-- Where this shape still wins: a rarely written total. In the deck's +-- illustrative model, embedding is cheaper below about 4.4 CDRs a subscriber a +-- day (light users, IoT SIMs, a closed cycle's final bill). +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @01-document-model.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables WHERE table_name = 'CP_SUBSCRIBER_DOC') LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' PURGE'; + END LOOP; +END; +/ + +CREATE JSON COLLECTION TABLE cp_subscriber_doc; + +-- S-001 mid-cycle: 1,000 prior CDR line items in usage[] (~70 KB of JSON), with +-- the baked cycleUsage rollup equal to their sums. Generated, not pasted. +INSERT INTO cp_subscriber_doc +SELECT JSON_OBJECT('_id' VALUE 'S-001', 'msisdn' VALUE '+1-202-555-0111', 'plan' VALUE 'UNLIMITED', + 'cycleUsage' VALUE JSON_OBJECT('totalMB' VALUE total_mb, 'totalMin' VALUE total_min, 'cost' VALUE total_cost), + 'usage' VALUE usage RETURNING JSON) +FROM (SELECT SUM(mb) AS total_mb, SUM(mins) AS total_min, SUM(cost) AS total_cost, + JSON_ARRAYAGG(JSON_OBJECT('cdrId' VALUE n, 'ts' VALUE ts, 'mb' VALUE mb, 'min' VALUE mins, 'cost' VALUE cost) + ORDER BY n RETURNING JSON) AS usage + FROM (SELECT LEVEL AS n, + 50 + MOD(LEVEL * 37, 200) AS mb, + 1 + MOD(LEVEL * 13, 30) AS mins, + ROUND((1 + MOD(LEVEL * 13, 30)) * 0.03, 2) AS cost, + TIMESTAMP '2026-09-01 00:00:00' + NUMTODSINTERVAL(LEVEL * 40, 'MINUTE') AS ts + FROM dual CONNECT BY LEVEL <= 1000)); +-- S-002 and S-003: one CDR each. Built the same way as S-001, so usage[].ts is +-- the same ISO-8601 string (no zone) in every document and in the appended CDR. +INSERT INTO cp_subscriber_doc +SELECT JSON_OBJECT('_id' VALUE 'S-002', 'msisdn' VALUE '+1-202-555-0122', 'plan' VALUE 'METERED', + 'cycleUsage' VALUE JSON_OBJECT('totalMB' VALUE 50, 'totalMin' VALUE 5, 'cost' VALUE 0.15), + 'usage' VALUE JSON_ARRAY(JSON_OBJECT('cdrId' VALUE 1, 'ts' VALUE TIMESTAMP '2026-09-02 09:00:00', + 'mb' VALUE 50, 'min' VALUE 5, 'cost' VALUE 0.15) RETURNING JSON) + RETURNING JSON) +FROM dual; +INSERT INTO cp_subscriber_doc +SELECT JSON_OBJECT('_id' VALUE 'S-003', 'msisdn' VALUE '+1-202-555-0133', 'plan' VALUE 'UNLIMITED', + 'cycleUsage' VALUE JSON_OBJECT('totalMB' VALUE 900, 'totalMin' VALUE 40, 'cost' VALUE 2.7), + 'usage' VALUE JSON_ARRAY(JSON_OBJECT('cdrId' VALUE 1, 'ts' VALUE TIMESTAMP '2026-09-03 14:00:00', + 'mb' VALUE 900, 'min' VALUE 40, 'cost' VALUE 2.7) RETURNING JSON) + RETURNING JSON) +FROM dual; +COMMIT; + +-- One CDR arrives for S-001. In the baked-rollup model this is a read-modify-WRITE +-- of the entire subscriber document: append the line item, re-tick the rollup. +-- Do this per CDR, thousands of times a cycle, on your busiest subscribers. +-- @step Post one Call Detail Record for S-001 +-- @note Append the line item and re-tick the rollup: a read-modify-write of the WHOLE subscriber document (1,000 prior CDRs), once per CDR. +-- @why Update locality is the hot knob here: every call lands on the same parent, so each CDR pays for rewriting everything already in the document. +-- @look Rows affected is 1, but Measure it shows how many bytes that one row cost. +-- @figure doc-shape.svg The subscriber document as built: a running cycle-usage rollup that every CDR rewrites +-- @measure record-cdr +UPDATE cp_subscriber_doc +SET data = JSON_TRANSFORM(data, + APPEND '$.usage' = JSON_OBJECT('cdrId' VALUE 1001, 'ts' VALUE TIMESTAMP '2026-09-29 18:00:00', + 'mb' VALUE 120, 'min' VALUE 12, 'cost' VALUE 0.36), + SET '$.cycleUsage.totalMB' = JSON_VALUE(data,'$.cycleUsage.totalMB' RETURNING NUMBER) + 120, + SET '$.cycleUsage.totalMin' = JSON_VALUE(data,'$.cycleUsage.totalMin' RETURNING NUMBER) + 12, + SET '$.cycleUsage.cost' = JSON_VALUE(data,'$.cycleUsage.cost' RETURNING NUMBER) + 0.36) +WHERE JSON_VALUE(data,'$._id') = 'S-001'; +COMMIT; +-- ^ every CDR => a rewrite of the whole, growing S-001 document. That is the write amplification. + +-- The account page read the pattern optimizes for (one document, already summed): +-- @step Read the account page (rollup already there) +-- @note One document, one lookup: the CDR's cost already landed on the write. Shows the line-item count, not the usage[] array itself. +-- @why Read/write is where the pattern earns its keep: 30 balance checks a subscriber a day read a precomputed rollup from one document. +-- @look mb_this_cycle comes straight off the document, and cdrs_in_document shows the 1,000-plus line items riding along with it. +SELECT JSON_VALUE(data,'$._id') AS subscriber, + JSON_VALUE(data,'$.cycleUsage.totalMB' RETURNING NUMBER) AS mb_this_cycle, + JSON_VALUE(data,'$.usage.size()' RETURNING NUMBER) AS cdrs_in_document +FROM cp_subscriber_doc +WHERE JSON_VALUE(data,'$._id') = 'S-001'; + +-- The Top-N dashboard read the pattern does NOT help (full scan + sort every load): +-- @step Rank the top talkers (full scan + sort) +-- @note No rollup to lean on across subscribers: SUM/SORT/LIMIT the whole collection, every load. +-- @why Diversity bites here: the ops dashboard ranks usage across subscribers, and a rollup inside each document gives it nothing to seek on, so every load sorts the whole collection. +-- @look Up to 5 subscribers ranked by mb_this_cycle, produced by sorting every document in the collection. +SELECT JSON_VALUE(data,'$._id') AS subscriber, + JSON_VALUE(data,'$.cycleUsage.totalMB' RETURNING NUMBER) AS mb_this_cycle +FROM cp_subscriber_doc +ORDER BY JSON_VALUE(data,'$.cycleUsage.totalMB' RETURNING NUMBER) DESC +FETCH FIRST 5 ROWS ONLY; diff --git a/labs/converged-modeling-patterns/patterns/02-computed/02-converged.sql b/labs/converged-modeling-patterns/patterns/02-computed/02-converged.sql new file mode 100644 index 0000000..8f90bae --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/02-computed/02-converged.sql @@ -0,0 +1,146 @@ +-- ============================================================================ +-- Pattern 02 · Computed · THE CONVERGED ALTERNATIVE +-- Two moves, both keeping the precomputed read without paying whole-document write amp. +-- +-- 1) CDRs are APPEND-ONLY relational rows: one small insert per CDR, no parent +-- document to rewrite. A maintained summary carries the rollup and is kept +-- current IN THE SAME TRANSACTION by a row trigger. Both models serialize per +-- subscriber (the summary row locks, just as the document does); the +-- converged write is ~1.5 KB instead of the whole document: +-- "Computed, with a staleness window of zero." (Declarative equivalent: +-- a materialized view WITH REFRESH FAST ON COMMIT over a CDR mview log.) +-- +-- 2) Hot Top-N becomes an INDEX RANGE SCAN. A descending index on the summary's +-- usage column turns "top talkers" into an O(log n) seek + FETCH FIRST N, +-- no SUM, no SORT, no full-collection scan. This is the field pattern behind +-- the 60s -> 500ms dashboard result (a real landing page, anonymized). +-- +-- The needle-flip, resolved: the read is still one precomputed rollup, now +-- joined by primary key (about 30% more read work in the deck's model, paid on +-- every balance check); the write drops from a rewrite of the whole, growing +-- document to one small row plus a single-row counter update, flat however many +-- CDRs the cycle holds; the ranking stops being a scan. The CBO plans all of it, +-- ACID, one transaction. +-- +-- When to flip back: a rarely written total (light subscribers below about 4.4 +-- CDRs a day, a closed cycle's final bill) can live on the document. +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @02-converged.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables + WHERE table_name IN ('CP_CDR','CP_SUBSCRIBER_USAGE','CP_SUBSCRIBERS') + ORDER BY table_name) LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' CASCADE CONSTRAINTS PURGE'; + END LOOP; +END; +/ + +CREATE TABLE cp_subscribers ( + subscriber_id VARCHAR2(12) PRIMARY KEY, + msisdn VARCHAR2(24) NOT NULL, + plan VARCHAR2(16) NOT NULL +); + +-- Append-only fact: one small row per CDR. No document to rewrite. +CREATE TABLE cp_cdr ( + cdr_id NUMBER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + subscriber_id VARCHAR2(12) NOT NULL REFERENCES cp_subscribers(subscriber_id), + mb NUMBER(12,2) DEFAULT 0 NOT NULL, + minutes NUMBER(10) DEFAULT 0 NOT NULL, + cost NUMBER(12,4) DEFAULT 0 NOT NULL, + cdr_ts TIMESTAMP DEFAULT SYSTIMESTAMP NOT NULL +); + +-- Maintained rollup: "Computed with staleness = 0." +CREATE TABLE cp_subscriber_usage ( + subscriber_id VARCHAR2(12) PRIMARY KEY REFERENCES cp_subscribers(subscriber_id), + total_mb NUMBER(14,2) DEFAULT 0 NOT NULL, + total_min NUMBER(12) DEFAULT 0 NOT NULL, + cost NUMBER(14,4) DEFAULT 0 NOT NULL +); + +-- The descending index that turns Top-N into a range scan (not a sort). +CREATE INDEX cp_ix_usage_topn ON cp_subscriber_usage (total_mb DESC, subscriber_id); + +CREATE OR REPLACE TRIGGER cp_trg_usage +AFTER INSERT ON cp_cdr +FOR EACH ROW +BEGIN + MERGE INTO cp_subscriber_usage u + USING (SELECT :NEW.subscriber_id AS sid FROM dual) s + ON (u.subscriber_id = s.sid) + WHEN MATCHED THEN UPDATE SET + u.total_mb = u.total_mb + :NEW.mb, + u.total_min = u.total_min + :NEW.minutes, + u.cost = u.cost + :NEW.cost + WHEN NOT MATCHED THEN INSERT (subscriber_id, total_mb, total_min, cost) + VALUES (:NEW.subscriber_id, :NEW.mb, :NEW.minutes, :NEW.cost); +END; +/ + +INSERT INTO cp_subscribers VALUES ('S-001','+1-202-555-0111','UNLIMITED'); +INSERT INTO cp_subscribers VALUES ('S-002','+1-202-555-0122','METERED'); +INSERT INTO cp_subscribers VALUES ('S-003','+1-202-555-0133','UNLIMITED'); + +-- The same mid-cycle state as the document model: 1,000 prior CDRs for S-001 +-- (the trigger builds the matching summary row), one each for S-002 and S-003. +INSERT INTO cp_cdr (subscriber_id, mb, minutes, cost, cdr_ts) +SELECT 'S-001', + 50 + MOD(LEVEL * 37, 200), + 1 + MOD(LEVEL * 13, 30), + ROUND((1 + MOD(LEVEL * 13, 30)) * 0.03, 2), + TIMESTAMP '2026-09-01 00:00:00' + NUMTODSINTERVAL(LEVEL * 40, 'MINUTE') +FROM dual CONNECT BY LEVEL <= 1000; +INSERT INTO cp_cdr (subscriber_id, mb, minutes, cost, cdr_ts) VALUES ('S-002', 50, 5,0.15, TIMESTAMP '2026-09-02 09:00:00'); +INSERT INTO cp_cdr (subscriber_id, mb, minutes, cost, cdr_ts) VALUES ('S-003',900,40,2.70, TIMESTAMP '2026-09-03 14:00:00'); +COMMIT; + +-- CDRs stream in as tiny appends; the summary tracks them in the same commit. +-- @step Post one Call Detail Record for S-001 +-- @note One small row; the trigger maintains the summary in the same transaction. Both models serialize per subscriber; the converged write is ~1.5 KB instead of the whole document. +-- @why Update locality still lands on one subscriber, but the write is a small CDR insert plus a one-row counter bump, not a rewrite of every prior CDR. Both models serialize per subscriber; here each serialized write is about 1.5 KB instead of the whole document. +-- @look Rows affected is 1; Measure it shows redo that stays near 1.6 KB however many CDRs the cycle holds. +-- @figure erd.svg Subscriber, CDR and usage summary in canonical form, beside the embedded rollup that concentrates every CDR on one document +-- @measure record-cdr +INSERT INTO cp_cdr (subscriber_id, mb, minutes, cost) VALUES ('S-001',120,12,0.36); +COMMIT; + +-- Account page read: the rollup is already there (staleness = 0), one PK lookup. +-- @step Read the account page (rollup already there) +-- @note Staleness = 0: the trigger kept this current in the same transaction as the CDR insert. +-- @why Read/write: the balance check still reads a precomputed rollup, now from a narrow summary row the trigger keeps current in the CDR's own transaction. +-- @look total_mb and cost come from one primary-key lookup on cp_subscriber_usage, with no aggregation over cp_cdr. +SELECT subscriber_id, total_mb, cost FROM cp_subscriber_usage WHERE subscriber_id = 'S-001'; + +-- Hot Top-N: an index range scan on cp_ix_usage_topn + FETCH FIRST N. +-- (EXPLAIN PLAN FOR this SELECT shows INDEX RANGE SCAN, not SORT ORDER BY.) +-- @step Rank the top talkers (index range scan) +-- @note No SUM, no SORT: the descending index turns Top-N into a range scan that stops after N rows. +-- @note Field result for this change (anonymized): the dashboard's hot Top-N went from 60 s to 500 ms. +-- @why Diversity: the ops dashboard gets its own access path, a descending index on total_mb, so Top-N stops after 5 index entries instead of sorting every subscriber. +-- @look Up to 5 rows in total_mb order, the same ranking as the document model's sort, read from the head of the index. +SELECT subscriber_id, total_mb, cost +FROM cp_subscriber_usage +ORDER BY total_mb DESC +FETCH FIRST 5 ROWS ONLY; + +-- Prefer the document shape at the API? Project it: rollup included, live. +CREATE OR REPLACE JSON RELATIONAL DUALITY VIEW cp_subscriber_dv AS +SELECT JSON { + '_id' : s.subscriber_id, + 'msisdn' : s.msisdn WITH UPDATE, + 'plan' : s.plan WITH UPDATE, + 'cycleUsage' : ( SELECT JSON { 'totalMB' : u.total_mb, + 'totalMin': u.total_min, + 'cost' : u.cost } + FROM cp_subscriber_usage u WITH NOUPDATE + WHERE u.subscriber_id = s.subscriber_id ) +} FROM cp_subscribers s WITH INSERT UPDATE DELETE; + +-- @step Read the projected subscriber document +-- @note The duality view includes the live rollup: no separate document to maintain for the API shape. +-- @why Diversity without a second copy: the duality view projects the subscriber document with cycleUsage inline and read-only, joined by primary key on each read. +-- @look subscriber_document shows cycleUsage with the same totals as the account-page read. +SELECT JSON_SERIALIZE(data PRETTY) AS subscriber_document +FROM cp_subscriber_dv WHERE JSON_VALUE(data,'$._id') = 'S-001'; diff --git a/labs/converged-modeling-patterns/patterns/02-computed/03-parity.js b/labs/converged-modeling-patterns/patterns/02-computed/03-parity.js new file mode 100644 index 0000000..e99cee2 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/02-computed/03-parity.js @@ -0,0 +1,42 @@ +// ============================================================================ +// Pattern 02 · Computed · CROSS-API PARITY (mongosh) +// The SAME subscriber document (with its live rollup) read through the SQL +// duality view (cp_subscriber_dv, injected as $SQL_RESULT) and through the MongoDB +// API. Byte-equal after canonicalizing JSON and ignoring _metadata. Non-zero exit +// on mismatch. +// ============================================================================ +function canon(x) { + if (x === null || x === undefined) return null; + if (Array.isArray(x)) return x.map(canon).sort((a, b) => + JSON.stringify(a) < JSON.stringify(b) ? -1 : 1); + if (typeof x === "object") { + const cn = x.constructor && x.constructor.name; + if (x._bsontype || /Decimal128|Double|Long|Int32/.test(cn || "")) return Number(x.toString()); + const o = {}; + for (const k of Object.keys(x).sort()) { if (k === "_metadata") continue; o[k] = canon(x[k]); } + return o; + } + return x; +} +const eq = (a, b) => JSON.stringify(canon(a)) === JSON.stringify(canon(b)); + +const viaSql = JSON.parse(process.env.SQL_RESULT || "null"); +if (!viaSql) { print("[FAIL] SQL_RESULT not provided by run.sh"); quit(1); } + +// @step Read the projected document over the MongoDB API +// @note Same subscriber document the SQL lane read: cycleUsage is the live rollup, whichever wire protocol you read it over. +// @why Diversity without copies: the self-care app keeps find() over the MongoDB API while charging reads the same rows as SQL/JSON, so there is one rollup and no document to keep in step. +// @look cycleUsage.totalMB in the returned document matches the SQL projection. +// @figure flow.svg Append-only CDRs maintain a usage row by trigger; one duality view serves the app over the MongoDB API and charging over SQL/JSON +const viaMongo = db.cp_subscriber_dv.findOne({ _id: "S-001" }); +if (!viaMongo) { print("[FAIL] MongoDB API returned no document for S-001"); quit(1); } + +if (!eq(viaSql, viaMongo)) { + print("[FAIL] cross-API document mismatch for subscriber S-001"); + print(" SQL lane: " + JSON.stringify(canon(viaSql))); + print(" Mongo lane: " + JSON.stringify(canon(viaMongo))); + quit(1); +} +print("[PASS] cp_subscriber_dv document byte-equal across SQL and MongoDB API (S-001)"); +print(" cycleUsage.totalMB via Mongo lane = " + viaMongo.cycleUsage.totalMB); +quit(0); diff --git a/labs/converged-modeling-patterns/patterns/02-computed/README.md b/labs/converged-modeling-patterns/patterns/02-computed/README.md new file mode 100644 index 0000000..2e8eb5a --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/02-computed/README.md @@ -0,0 +1,136 @@ +--- +title: Computed +industry: Telecom +deck: "18–21" +problem: >- + Each subscriber document carries a running cycle-usage rollup so the app and real-time + charging read one document, but every call record now rewrites that hot document. +knobs: + - { name: Diversity, setting: "Medium-high: 4 consumers", help: "The self-care app, real-time charging, the ops dashboard and the bill run all read usage, each in a different shape." } + - { name: Read / write, setting: "Write-heavy on the rollup", help: "About 30 balance reads a subscriber a day against 11 CDRs on average and 400-plus for heavy users, so on average the rollup is written once for every three reads, and for heavy users it is written more often than it is read." } + - { name: Update locality, setting: "Every write, one parent", hot: true, help: "Every CDR lands on the same subscriber document: break-even is about 4.4 CDRs a subscriber a day, and a 400-a-day heavy user costs about 2.5× the work embedded." } +help: + tabs: + document: + why: "The starting point: keep a running cycleUsage rollup on the subscriber document so the balance check reads one document. The read is right; the cost is that every CDR rewrites the whole, growing document." + look: "cdrs_in_document on the account page: the line items that every new CDR has to rewrite." + converged: + why: "CDRs become append-only rows and a trigger keeps a narrow usage row current in the same transaction. The balance read joins that row by primary key (about 30% more read work in the deck's model); each CDR is a small insert plus a one-row counter bump." + look: "The CDR insert reports 1 row affected, and the account page reads total_mb from one summary row." + mongo: + why: "The subscriber document the app wanted, projected over the MongoDB API with the live rollup inline and read-only." + look: "cycleUsage.totalMB in the returned document matches total_mb from the SQL reads." + measure: + why: "The ratio is the document model's redo for one CDR divided by the converged model's. The document side rewrites every line item already in the subscriber document, so the ratio climbs as the cycle fills; the converged insert stays near 1.6 KB." + look: "Your dot at 1,000 line items, near the 33× reference point (about 52 KB of redo against 1.6 KB)." +measure: + x_label: "CDR line items in the subscriber document" + lab_x: 1000 + deck_slides: "18–21" + calibration: + - { x: 10, ratio: 1.73 } # doc 2708 B, conv 1568 B + - { x: 100, ratio: 7.16 } # doc 11288 B, conv 1576 B + - { x: 1000, ratio: 32.53 } # doc 51792 B, conv 1592 B + - { x: 5000, ratio: 203.24 } # doc 323560 B, conv 1592 B +--- +# Pattern 02: Computed + +**Telecom.** Show a subscriber's current-cycle usage instantly on the account page, +and rank the top talkers for the ops dashboard. + +> Keep the read win. Stop paying for it with a whole-document rewrite on every event. + +## The document bet + +The Computed pattern bakes a rollup onto the document: compute once on write, then +serve every read from the stored answer instead of re-aggregating a month of CDRs. +The math is real: a million reads an hour against a thousand writes an hour means +precomputing "divides the work by a thousand." Store `cycleUsage` on the subscriber +and the self-care app and real-time charging each get the balance from a single +lookup. The read side of this bet is right; the question is what it costs on the +write side, and who pays. + +## Where the needle flips + +Usage is not written once. Every Call Detail Record (CDR) appends a line item and +re-ticks the rollup, and each one is a read-modify-write that **rewrites the whole +subscriber document**. The document grows with every call, so the redo each CDR +costs grows with it: in the lab, subscriber S-001 already carries 1,000 CDR line +items this cycle, and posting one more rewrites all of them. A heavy user throws off +400 or more CDRs a day, so the busiest subscribers own the hottest documents. The +pattern's escape hatch, counter sharding (split the counter into N shard documents), +does spread the write contention, but now every balance check reads N documents and +sums them: a write problem traded for a read problem, on the latency-critical +charging path. + +And the Top-N dashboard gets no help at all: with the rollup buried inside each of +12 million documents, ranking the top talkers means **SUM, SORT, LIMIT across the +whole collection on every load.** + +**Model the heavy tail, not the average.** In the deck's illustrative cost model +(30 balance reads a subscriber a day; a CDR rewrites the ~4 KB document, versus a +row insert plus a counter-row update), embedding wins below about **4.4 CDRs a +subscriber a day**. The average subscriber, at 11, is past break-even; the heavy +user at 400 costs about **2.5× the work** embedded, concentrated on exactly the +documents under contention. + +## The converged softening + +In canonical form (the logical model) a CDR is an immutable fact, appended once, and +the cycle rollup is one derived value per subscriber. The Computed pattern splits +into two things that were never the same: facts and a derivation. Two moves follow: + +1. **CDRs are append-only rows.** One small insert per event, no parent document to + rewrite. A narrow summary row carries the rollup and is kept current **in the + same transaction** by a row trigger: *Computed, with a staleness window of zero.* + (Declarative equivalent: a materialized view `REFRESH FAST ON COMMIT` over a CDR + mview log.) Both models serialize per subscriber; the summary row locks just as + the document does. What changes is what you do while holding the lock: a ~1.5 KB + write instead of a rewrite of the whole, growing document. +2. **Hot Top-N becomes an index range scan.** A descending index on the summary's + usage column turns "top talkers" into an O(log n) seek plus `FETCH FIRST N`, no + SUM, no SORT, no full-collection scan. + +This is the shape behind an anonymized field result on exactly this workload: a +landing-page Top-N that went **from 60 seconds to 500 milliseconds**, ACID, with no +precompute job, no change streams, and no eventual-consistency window. Want the +document shape at the API? A duality view projects the subscriber with the live +rollup inline and read-only, through the same privileges, auditing and policies as +the tables underneath. + +Measured in this lab on 26ai Free, recording one CDR against a subscriber document +already holding 1,000 CDRs costs about 52 KB of redo; the converged insert plus the +trigger-maintained summary row costs about 1.6 KB, roughly **33×** less. + +## The tradeoff, honestly + +| | Embedded rollup | Append + summary | +|---|---|---| +| Record a CDR | whole-document rewrite (~52 KB redo at 1,000 CDRs) | small insert + one counter row (~1.6 KB) | +| Balance read | 1 document | 1 document, joined by primary key (~30% more read work) | +| Top 20 talkers | sort every document | index range scan | + +The read premium is real: every balance check pays it, because the projection joins +the usage row on each read. What it buys is the end of the whole-document rewrite on +every CDR. + +**When the document shape is still right:** a rarely written total can live on the +document. Light subscribers and IoT SIMs below the ~4 CDRs-a-day break-even, or a +closed cycle's final bill, are cheaper with the embedded rollup. Compute-on-write +belongs on the document when writes are rare and don't collide. + +## In this folder + +| File | What it shows | +|---|---| +| `01-document-model.sql` | Baked `cycleUsage` over 1,000 embedded CDR line items; each CDR rewrites the whole, growing subscriber doc; Top-N is a full scan | +| `02-converged.sql` | The same 1,000 CDRs as append-only rows + trigger-maintained summary (staleness 0) + descending Top-N index + duality projection | +| `03-parity.js` | Reads `cp_subscriber_dv` (rollup included) via the MongoDB API and asserts byte-equality with the SQL lane | +| `_capture.sql` | Helper: emits the SQL-lane document for the parity check | + +## Validated + +Both lanes run clean on **Oracle AI Database 26ai Free (`23.26.3-faststart`)**. Cross-API +parity passes: the subscriber document with its live rollup is identical through SQL and +the MongoDB API. +Run with `../../run.sh 02-computed`. diff --git a/labs/converged-modeling-patterns/patterns/02-computed/_capture.sql b/labs/converged-modeling-patterns/patterns/02-computed/_capture.sql new file mode 100644 index 0000000..f309ae4 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/02-computed/_capture.sql @@ -0,0 +1,7 @@ +-- Capture the subscriber document as seen from SQL (duality view cp_subscriber_dv), +-- serialized to one JSON string, for cross-API parity against the MongoDB API. +SET HEADING OFF PAGESIZE 0 FEEDBACK OFF VERIFY OFF TRIMSPOOL ON +SET LINESIZE 32767 LONG 100000000 LONGCHUNKSIZE 1000000 +SELECT JSON_SERIALIZE(data RETURNING CLOB) +FROM cp_subscriber_dv +WHERE JSON_VALUE(data, '$._id') = 'S-001'; diff --git a/labs/converged-modeling-patterns/patterns/02-computed/calibrate.sql b/labs/converged-modeling-patterns/patterns/02-computed/calibrate.sql new file mode 100644 index 0000000..d2ebab5 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/02-computed/calibrate.sql @@ -0,0 +1,28 @@ +-- Calibration resize for Measure it: subscriber S-001 carries :n CDR line items on BOTH +-- sides, generated exactly as the lab's setup generates its 1,000. +-- Document model: rebuild the S-001 document with :n usage[] items and a matching rollup. +DELETE FROM cp_subscriber_doc WHERE JSON_VALUE(data, '$._id') = 'S-001'; +INSERT INTO cp_subscriber_doc +SELECT JSON_OBJECT('_id' VALUE 'S-001', 'msisdn' VALUE '+1-202-555-0111', 'plan' VALUE 'UNLIMITED', + 'cycleUsage' VALUE JSON_OBJECT('totalMB' VALUE total_mb, 'totalMin' VALUE total_min, 'cost' VALUE total_cost), + 'usage' VALUE usage RETURNING JSON) +FROM (SELECT SUM(mb) AS total_mb, SUM(mins) AS total_min, SUM(cost) AS total_cost, + JSON_ARRAYAGG(JSON_OBJECT('cdrId' VALUE k, 'ts' VALUE ts, 'mb' VALUE mb, 'min' VALUE mins, 'cost' VALUE cost) + ORDER BY k RETURNING JSON) AS usage + FROM (SELECT LEVEL AS k, + 50 + MOD(LEVEL * 37, 200) AS mb, + 1 + MOD(LEVEL * 13, 30) AS mins, + ROUND((1 + MOD(LEVEL * 13, 30)) * 0.03, 2) AS cost, + TIMESTAMP '2026-09-01 00:00:00' + NUMTODSINTERVAL(LEVEL * 40, 'MINUTE') AS ts + FROM dual CONNECT BY LEVEL <= :n)); +-- Converged: :n CDR rows for S-001; zero the summary first so the trigger rebuilds it. +DELETE FROM cp_cdr WHERE subscriber_id = 'S-001'; +UPDATE cp_subscriber_usage SET total_mb = 0, total_min = 0, cost = 0 WHERE subscriber_id = 'S-001'; +INSERT INTO cp_cdr (subscriber_id, mb, minutes, cost, cdr_ts) +SELECT 'S-001', + 50 + MOD(LEVEL * 37, 200), + 1 + MOD(LEVEL * 13, 30), + ROUND((1 + MOD(LEVEL * 13, 30)) * 0.03, 2), + TIMESTAMP '2026-09-01 00:00:00' + NUMTODSINTERVAL(LEVEL * 40, 'MINUTE') +FROM dual CONNECT BY LEVEL <= :n; +COMMIT; diff --git a/labs/converged-modeling-patterns/patterns/02-computed/figures/doc-shape.svg b/labs/converged-modeling-patterns/patterns/02-computed/figures/doc-shape.svg new file mode 100644 index 0000000..e89ba38 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/02-computed/figures/doc-shape.svg @@ -0,0 +1 @@ +subscribers · one document per subscriber{"_id": "S-4471902","msisdn": "+1-202-555-0142","plan": "UNLIMITED-5G","addOns": [ "roam-EU", "hotspot-50" ],"devices": [ { "imei": "35-2099", ... } ],"cycle": "2026-09","cycleUsage": {"totalMB": 48213.7,"totalMin": 1284,"cost": 86.42,"cdrCount": 11906},"updatedAt": "2026-09-28T14:02:11Z"}every CDR rewritesthe whole document:400+ times a dayTop-N sorts 12Mdocuments by thisburied field \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/02-computed/figures/erd.svg b/labs/converged-modeling-patterns/patterns/02-computed/figures/erd.svg new file mode 100644 index 0000000..c6ad51b --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/02-computed/figures/erd.svg @@ -0,0 +1 @@ +CANONICAL FORM: FACTS APPENDED, ROLLUP DERIVEDTHE SAME ROLLUP, EMBEDDEDSUBSCRIBERPKsubscriber_idvarchar2msisdnvarchar2planvarchar2cyclevarchar2CDR · append-onlyPKcdr_idnumberFKsubscriber_idvarchar2mbnumberminutesnumbercostnumbercdr_tstimestampSUBSCRIBER_USAGEPKsubscriber_idvarchar2total_mbnumbertotal_minnumbercostnumberAFTER INSERT trigger+1 counter row, same txnCDR stream1,500/s avg · 6k/s peakS-4471902 · one hot document400+ whole-doc rewrites / dayshard 1shard 2shard 3shard Nbalance read = N reads + SUMa write problem traded for a read oneRelational projectionappend 1 row + bump 1counter row · atomicmutable rollup: every CDR writes itimmutable CDR facts: append, never rewrite \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/02-computed/figures/flow.svg b/labs/converged-modeling-patterns/patterns/02-computed/figures/flow.svg new file mode 100644 index 0000000..051f2b9 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/02-computed/figures/flow.svg @@ -0,0 +1 @@ +RELATIONAL PROJECTIONMAINTAINED · PROJECTEDCONSUMERScdrappend-only rowssubscriber_usagethe rollup · 1 rowsubscribersmsisdn · planix_usage_topntotal_mb DESCsubscriber_dvduality viewcycleUsage NOUPDATEBill runSQL over cdrOps Top-Nindex range scanSelf-care appMongo API find()ChargingSQL/JSONtriggersame txn \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/02-computed/figures/model-curve.svg b/labs/converged-modeling-patterns/patterns/02-computed/figures/model-curve.svg new file mode 100644 index 0000000..85d5fb5 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/02-computed/figures/model-curve.svg @@ -0,0 +1 @@ +KNOB SETTINGS · THIS WORKLOAD1 · DiversityMedium-high: 4 consumersapp, charging, ops dashboardand the bill run2 · Read / writeWrite-heavy on the rollup~30 balance reads/day vs11 CDRs avg · 400+ heavy3 · Update localityEvery write, one parenteach CDR lands on the samehot subscriber documentWORK / SUBSCRIBER / DAY vs CDRs / DAY · illustrative modelembed winsappend + summary is cheaper1003001k3k10k1101001kCDRs per subscriber per day (log)work units / subscriber / day (log)break-even ≈ 4.4 / daylight · 1.5average · 11heavy · 400Embedded rollup · doc rewriteAppend + summary · +30% read \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/01-document-model.js b/labs/converged-modeling-patterns/patterns/03-bucket/01-document-model.js new file mode 100644 index 0000000..b49a65c --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/01-document-model.js @@ -0,0 +1,41 @@ +// ============================================================================ +// Pattern 03 · Bucket · THE DOCUMENT MODEL through the MongoDB API (mongosh) +// The $push bucket: each reading grows the bucket document and re-updates its +// counters: a read-modify-write of the WHOLE (growing) document on every +// append, the textbook write-amplification. Runnable illustration. +// Run via run.sh (Mongo lane) or: mongosh "" --file 01-document-model.js +// ============================================================================ +const C = db.bk_bucket_js; +C.drop(); + +// One bucket per machine per hour; starts empty. +C.insertOne({ _id: "M-100|TEMP|2026-08-01T10", machineId: "M-100", metric: "TEMP", + hourStart: "2026-08-01T10:00:00Z", count: 0, sum: 0, max: null, readings: [] }); + +// Readings arrive. Each one is a $push + counter update: one whole-document +// rewrite per reading. On a hot sensor this bucket marches at the 16 MB ceiling. +const arrivals = [ + { ts: "2026-08-01T10:00:05Z", val: 88.4 }, + { ts: "2026-08-01T10:00:10Z", val: 89.1 }, + { ts: "2026-08-01T10:00:15Z", val: 91.7 } +]; +for (const r of arrivals) { + C.updateOne({ _id: "M-100|TEMP|2026-08-01T10" }, + { $push: { readings: { ts: r.ts, val: r.val } }, + $inc: { count: 1, sum: r.val }, + $max: { max: r.val } }); +} + +const b = C.findOne({ _id: "M-100|TEMP|2026-08-01T10" }); +print(`bucket after ${b.count} appends: sum=${b.sum} max=${b.max} ` + + `(each append rewrote the whole document)`); + +// Sanity: the maintained counters reflect the three readings. +const okCount = b.count === 3; +const okMax = Math.abs(b.max - 91.7) < 1e-9; +const okSum = Math.abs(b.sum - 269.2) < 1e-6; +if (!(okCount && okMax && okSum)) { + print(`[FAIL] bucket counters wrong: count=${b.count} sum=${b.sum} max=${b.max}`); + quit(1); +} +print("[OK] document-model bucket built (write-amp illustration)"); diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/01-document-model.sql b/labs/converged-modeling-patterns/patterns/03-bucket/01-document-model.sql new file mode 100644 index 0000000..0ca030b --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/01-document-model.sql @@ -0,0 +1,75 @@ +-- ============================================================================ +-- Pattern 03 · Bucket (time-series) · THE DOCUMENT MODEL (the starting point) +-- Industry: manufacturing / IoT. Access pattern: ingest millions of tiny sensor +-- readings; read back per-machine, per-hour summaries. +-- +-- The document move: don't store a document per reading (millions of tiny docs). +-- Group them into a bucket doc per sensor per hour, push each reading into an +-- array, and keep running counters on the bucket. Fewer, fatter documents, and a +-- one-document dashboard read. The read side is genuinely good. +-- +-- THE WRITE-AMPLIFICATION COST -- this is THE textbook write-amp pattern. Each +-- reading is an array $push PLUS a counter re-update: a read-modify-write of the +-- WHOLE bucket document, on every single append. The bucket only grows, so the +-- redo each append costs grows with it all cycle -- and a hot sensor marches its bucket +-- straight at the 16 MB document ceiling, where writes simply start failing. +-- The readings themselves are immutable facts; the amplification comes from the +-- mutable bucket wrapped around them. Slow sensors (below ~900 readings a bucket, +-- e.g. one a minute) are where this shape still wins. +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @01-document-model.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables WHERE table_name = 'BK_SENSOR_DOC') LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' PURGE'; + END LOOP; +END; +/ + +CREATE JSON COLLECTION TABLE bk_sensor_doc; + +-- One bucket: machine M-100, metric TEMP, hour starting 10:00. Starts small. +INSERT INTO bk_sensor_doc VALUES (JSON('{"_id":"M-100|TEMP|2026-08-01T10", + "machineId":"M-100","metric":"TEMP","hourStart":"2026-08-01T10:00:00Z", + "count":0,"sum":0,"max":null,"readings":[]}')); +COMMIT; + +-- Three readings arrive. Each one PUSHES to the array AND re-updates the +-- counters -- each one a read-modify-write of the whole (growing) bucket document. +-- @step Ingest three readings into the bucket +-- @note Each reading is an array APPEND plus a counter update: a read-modify-write of the WHOLE (growing) bucket, once per reading. +-- @why Update locality is the hot knob: every reading lands on the same bucket, so each append is a read-modify-write of every reading already in it. Bytes moved per hour grow with the square of the readings. +-- @look The PL/SQL block completes with no row count; Measure it shows what its 3 whole-bucket rewrites cost next to 3 row inserts. +-- @figure doc-shape.svg The sensor-hour bucket as built: counters and an array that every reading rewrites +-- @measure ingest-reading +BEGIN + FOR v IN (SELECT * FROM ( + SELECT 88.4 val, '2026-08-01T10:00:05Z' ts FROM dual UNION ALL + SELECT 89.1 val, '2026-08-01T10:00:10Z' ts FROM dual UNION ALL + SELECT 91.7 val, '2026-08-01T10:00:15Z' ts FROM dual)) LOOP + UPDATE bk_sensor_doc + SET data = JSON_TRANSFORM(data, + APPEND '$.readings' = JSON_OBJECT('ts' VALUE v.ts, 'val' VALUE v.val), + SET '$.count' = JSON_VALUE(data,'$.count' RETURNING NUMBER) + 1, + SET '$.sum' = JSON_VALUE(data,'$.sum' RETURNING NUMBER) + v.val, + SET '$.max' = GREATEST(NVL(JSON_VALUE(data,'$.max' RETURNING NUMBER), v.val), v.val)) + WHERE JSON_VALUE(data,'$._id') = 'M-100|TEMP|2026-08-01T10'; + END LOOP; +END; +/ +COMMIT; +-- ^ 3 whole-document rewrites for 3 readings. At real cadence the bucket is +-- rewritten on every tick and only gets more expensive as it fills. + +-- The read the pattern optimizes for: one bucket, counters already there. +-- @step Read the bucket (counters already there) +-- @note One document, running counters already maintained: cheap, until the bucket has to grow again. +-- @why Read/write is the bucket's case: a dashboard that reads the hour 360 times gets one contiguous document with the counters already computed. Below about 900 readings a bucket, that read win beats the rewrites. +-- @look n, avg_temp and max_temp come straight off the counters, with no pass over the readings array. +SELECT JSON_VALUE(data,'$.machineId') AS machine, + JSON_VALUE(data,'$.count' RETURNING NUMBER) AS n, + ROUND(JSON_VALUE(data,'$.sum' RETURNING NUMBER) + / JSON_VALUE(data,'$.count' RETURNING NUMBER), 3) AS avg_temp, + JSON_VALUE(data,'$.max' RETURNING NUMBER) AS max_temp +FROM bk_sensor_doc +WHERE JSON_VALUE(data,'$._id') = 'M-100|TEMP|2026-08-01T10'; diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/02-converged.sql b/labs/converged-modeling-patterns/patterns/03-bucket/02-converged.sql new file mode 100644 index 0000000..b0fbbb8 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/02-converged.sql @@ -0,0 +1,97 @@ +-- ============================================================================ +-- Pattern 03 · Bucket (time-series) · THE CONVERGED ALTERNATIVE +-- Stop hand-rolling buckets. Let range partitioning do the amortization the +-- bucket was faking, and let GROUP BY do the rollup. +-- +-- Readings are tiny, append-only rows in an INTERVAL-partitioned table -- Oracle +-- opens a new hourly partition automatically as time advances. Each reading is a +-- single small row: no array to grow, no bucket document to rewrite, no 16 MB +-- ceiling. The per-hour summary is a GROUP BY over a partition range (the +-- optimizer prunes to just the hours asked for). Want it precomputed? A +-- materialized view rolls it up; want it through the MongoDB API? A $sql stage +-- (Oracle's addition to the aggregation pipeline) runs the same GROUP BY with +-- parallel execution and NO 100 MB stage / 16 MB output caps (see README). +-- +-- The needle-flip, resolved: each reading is a constant-cost single-row insert +-- instead of an ever-growing document rewrite, and there is no document ceiling +-- to march toward. The price: the last-hour dashboard scans rows instead of +-- fetching one document, about 2x the read bytes in the deck's model. An +-- anonymized field result for exactly this reshape: a 29s bucket rollup fell to +-- under 400ms, paid for with that heavier last-hour read. Slow sensors (below +-- ~900 readings a bucket) are still better served by the bucket. +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @02-converged.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables WHERE table_name = 'BK_SENSOR_READINGS') LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' PURGE'; + END LOOP; +END; +/ + +-- Append-only, INTERVAL-partitioned by the hour. New partitions appear on their +-- own as readings cross hour boundaries -- the amortization the bucket faked, +-- done by the engine. +CREATE TABLE bk_sensor_readings ( + reading_id NUMBER GENERATED BY DEFAULT AS IDENTITY, + machine_id VARCHAR2(16) NOT NULL, + metric VARCHAR2(24) NOT NULL, + reading_val NUMBER(12,3) NOT NULL, + reading_ts TIMESTAMP NOT NULL +) +PARTITION BY RANGE (reading_ts) +INTERVAL (INTERVAL '1' HOUR) +( PARTITION p_seed VALUES LESS THAN (TIMESTAMP '2026-08-01 00:00:00') ); + +CREATE INDEX bk_ix_readings ON bk_sensor_readings (machine_id, metric, reading_ts); + +-- Every reading is one small row. No parent to rewrite; constant cost per reading. +-- The same three readings the document model appends to its bucket, as three rows. +-- @step Insert the three readings +-- @note Three small rows: no bucket to grow, no parent document to rewrite. +-- @why Update locality stops compounding: each reading is its own small row in the hour's partition, so an insert costs the same at reading 3 or reading 36,000. There is no document to grow and no 16 MB cap to reach. +-- @look Rows affected is 3; Measure it shows redo that stays near 1 KB at any bucket size. +-- @figure erd.svg Plant, machine and append-only readings with a derived hourly rollup, beside one hot sensor-hour packed into a single bucket +-- @measure ingest-reading +INSERT INTO bk_sensor_readings (machine_id, metric, reading_val, reading_ts) +SELECT 'M-100', 'TEMP', 88.4, TIMESTAMP '2026-08-01 10:00:05' FROM dual UNION ALL +SELECT 'M-100', 'TEMP', 89.1, TIMESTAMP '2026-08-01 10:00:10' FROM dual UNION ALL +SELECT 'M-100', 'TEMP', 91.7, TIMESTAMP '2026-08-01 10:00:15' FROM dual; +INSERT INTO bk_sensor_readings (machine_id, metric, reading_val, reading_ts) VALUES ('M-100','TEMP',80.2, TIMESTAMP '2026-08-01 11:03:00'); +INSERT INTO bk_sensor_readings (machine_id, metric, reading_val, reading_ts) VALUES ('M-200','TEMP',60.0, TIMESTAMP '2026-08-01 10:30:00'); +COMMIT; + +-- The rollup = GROUP BY over a partitioned range. Same answer the bucket +-- counters gave, computed on read, nothing maintained on write. Asking for a +-- single hour prunes to a single partition. +-- @step Roll up the hour with GROUP BY +-- @note Same answer the bucket counters gave, computed on read instead of maintained on every write. +-- @why Read/write, priced honestly: the rollup is computed on read and pruned to the partitions asked for, but a range scan pays per row where the bucket was one fetch. Where reads repeat, the materialized view precomputes it. +-- @look One row per machine and hour with n, avg_val and max_val, computed from the rows rather than maintained on every write. +SELECT machine_id, metric, TRUNC(reading_ts, 'HH24') AS hour_start, + COUNT(*) AS n, + ROUND(AVG(reading_val), 3) AS avg_val, + MAX(reading_val) AS max_val +FROM bk_sensor_readings +GROUP BY machine_id, metric, TRUNC(reading_ts, 'HH24') +ORDER BY machine_id, metric, hour_start; + +-- Precompute it if reads dwarf writes: a summary MV over the same rows. (Add +-- REFRESH FAST ON COMMIT + a materialized view log to make it staleness = 0.) +BEGIN + FOR r IN (SELECT mview_name FROM user_mviews WHERE mview_name = 'BK_HOURLY_MV') LOOP + EXECUTE IMMEDIATE 'DROP MATERIALIZED VIEW ' || r.mview_name; + END LOOP; +END; +/ +CREATE MATERIALIZED VIEW bk_hourly_mv +BUILD IMMEDIATE REFRESH COMPLETE ON DEMAND AS +SELECT machine_id, metric, TRUNC(reading_ts, 'HH24') AS hour_start, + COUNT(*) AS n, SUM(reading_val) AS sum_val, MAX(reading_val) AS max_val +FROM bk_sensor_readings +GROUP BY machine_id, metric, TRUNC(reading_ts, 'HH24'); + +SELECT machine_id, metric, hour_start, n, + ROUND(sum_val/n, 3) AS avg_val, max_val +FROM bk_hourly_mv +ORDER BY machine_id, metric, hour_start; diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/02-sql-in-pipeline.js b/labs/converged-modeling-patterns/patterns/03-bucket/02-sql-in-pipeline.js new file mode 100644 index 0000000..2c3a262 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/02-sql-in-pipeline.js @@ -0,0 +1,34 @@ +// ============================================================================ +// Pattern 03 · Bucket · $sql-IN-PIPELINE through the MongoDB API (mongosh) +// The converged hourly rollup, issued as FULL SQL over the Mongo wire protocol via +// $sql, Oracle's addition to the MongoDB aggregation pipeline (a full SQL statement +// as one stage of db.aggregate()): parallel execution, cost-based optimization, +// and NO 100 MB stage / 16 MB output caps. It reads the converged, INTERVAL- +// partitioned bk_sensor_readings table (created by the SQL lane's 02-converged.sql). +// This is the MongoDB API developer's on-ramp to the converged rollup: no leaving +// the pipeline, no maintaining a bucket. +// ============================================================================ +// @step Hourly rollup with $sql in the pipeline +// @note Full SQL over the Mongo wire protocol: parallel execution, cost-based optimization, no 100 MB stage / 16 MB output caps. +// @why Diversity: the anomaly scan keeps its pipeline over the MongoDB API, and the $sql stage hands the rollup to the optimizer with parallel execution and none of the pipeline's stage memory or output caps. +// @look One document per machine and hour with n, avg and max, the same values the SQL GROUP BY returns. +// @figure flow.svg Partitioned readings feed a pruned GROUP BY and a summary view, serving dashboards over SQL and the anomaly job through a $sql stage +const rows = db.aggregate([{ $sql: ` + select json { + 'machineId' : machine_id, + 'metric' : metric, + 'hourStart' : to_char(trunc(reading_ts,'HH24'),'YYYY-MM-DD"T"HH24'), + 'n' : count(*), + 'avg' : round(avg(reading_val),3), + 'max' : max(reading_val) + } + from bk_sensor_readings + group by machine_id, metric, trunc(reading_ts,'HH24') + order by machine_id, metric, trunc(reading_ts,'HH24')` }]).toArray(); + +print("hourly rollup via $sql-in-pipeline (rows returned over the Mongo wire):"); +for (const r of rows) { + print(` ${r.machineId} ${r.metric} ${r.hourStart} n=${r.n} avg=${r.avg} max=${r.max}`); +} +if (rows.length === 0) { print("[FAIL] $sql rollup returned no rows"); quit(1); } +print(`[OK] $sql-in-pipeline returned ${rows.length} rollup rows`); diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/03-parity.js b/labs/converged-modeling-patterns/patterns/03-bucket/03-parity.js new file mode 100644 index 0000000..b73dbe9 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/03-parity.js @@ -0,0 +1,64 @@ +// ============================================================================ +// Pattern 03 · Bucket · CROSS-LANE PARITY (mongosh) +// Proves the hourly rollup is identical whether issued through SQL*Plus (the SQL +// lane's 02-converged.sql GROUP BY, injected as $SQL_RESULT by run.sh) or through +// the Oracle API for MongoDB via a $sql-in-pipeline stage. Same machine/hour, same +// COUNT/AVG/MAX. Exits non-zero on any mismatch. +// ============================================================================ + +// --- canonicalize: sort keys, sort arrays, coerce BSON numerics, drop _metadata +function canon(x) { + if (x === null || x === undefined) return null; + if (Array.isArray(x)) return x.map(canon).sort((a, b) => + JSON.stringify(a) < JSON.stringify(b) ? -1 : 1); + if (typeof x === "object") { + const cn = x.constructor && x.constructor.name; + if (x._bsontype || /Decimal128|Double|Long|Int32/.test(cn || "")) return Number(x.toString()); + const o = {}; + for (const k of Object.keys(x).sort()) { if (k === "_metadata") continue; o[k] = canon(x[k]); } + return o; + } + if (typeof x === "number") return x; + return x; +} +const eq = (a, b) => JSON.stringify(canon(a)) === JSON.stringify(canon(b)); + +// SQL lane (from sqlplus, via run.sh) vs Mongo lane ($sql over the wire). +const sqlResult = JSON.parse(process.env.SQL_RESULT || "null"); +if (!sqlResult) { print("[FAIL] SQL_RESULT not provided by run.sh"); quit(1); } + +const mongoResult = db.aggregate([{ $sql: ` + select json { + 'machineId' : machine_id, + 'metric' : metric, + 'hourStart' : to_char(trunc(reading_ts,'HH24'),'YYYY-MM-DD"T"HH24'), + 'n' : count(*), + 'avg' : round(avg(reading_val),3), + 'max' : max(reading_val) + } + from bk_sensor_readings + group by machine_id, metric, trunc(reading_ts,'HH24') + order by machine_id, metric, trunc(reading_ts,'HH24')` }]).toArray(); + +if (!eq(sqlResult, mongoResult)) { + print("[FAIL] rollup parity mismatch"); + print(" SQL lane: " + JSON.stringify(canon(sqlResult))); + print(" Mongo lane: " + JSON.stringify(canon(mongoResult))); + quit(1); +} +print(`[PASS] rollup parity: SQL GROUP BY == Mongo $sql (${mongoResult.length} rows, COUNT/AVG/MAX equal)`); + +// Extra signal: the converged rollup for M-100/hour-10 equals the hand-maintained +// document-model bucket counters (from 01-document-model.js). +const bucket = db.bk_bucket_js.findOne({ _id: "M-100|TEMP|2026-08-01T10" }); +if (bucket) { + const cell = mongoResult.find(r => r.machineId === "M-100" && r.hourStart === "2026-08-01T10"); + const bAvg = Math.round((bucket.sum / bucket.count) * 1000) / 1000; + const okN = bucket.count === cell.n, okA = Math.abs(bAvg - cell.avg) < 1e-9, okM = Math.abs(bucket.max - cell.max) < 1e-9; + if (!(okN && okA && okM)) { + print(`[FAIL] bucket vs converged mismatch: bucket{n:${bucket.count},avg:${bAvg},max:${bucket.max}} converged{n:${cell.n},avg:${cell.avg},max:${cell.max}}`); + quit(1); + } + print("[PASS] maintained bucket counters == converged GROUP BY (M-100/hour-10)"); +} +quit(0); diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/README.md b/labs/converged-modeling-patterns/patterns/03-bucket/README.md new file mode 100644 index 0000000..a87579a --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/README.md @@ -0,0 +1,151 @@ +--- +title: Bucket +industry: Manufacturing / IoT +deck: "22–25" +problem: >- + Sensor readings are bucketed per sensor per hour. Every reading rewrites the + growing bucket (36,000 rewrites an hour on a hot spindle), and hot sensors march + it toward the 16 MB document cap. +knobs: + - { name: Diversity, setting: "Medium: 4 read shapes", help: "The live dashboard, the 8-hour shift rollup, the 7-day anomaly scan and ML feature pulls all read the same readings, each in a different shape." } + - { name: Read / write, setting: "Write-heavy", help: "About 34,000 readings a second, three billion a day, against a dashboard that re-reads each sensor-hour about 360 times." } + - { name: Update locality, setting: "Every write, same document", hot: true, help: "Every reading lands on the same growing bucket: break-even is about 900 readings a bucket, a one-a-second sensor already costs about 2.7× the rows, and a ten-a-second spindle about 22×, marching at the 16 MB cap. Slow sensors (one a minute, 60 a bucket) still favor the bucket." } +help: + tabs: + document: + why: "The starting point: one bucket document per sensor per hour, readings pushed into an array under running count, sum and max. The dashboard reads one document; every reading rewrites the growing bucket." + look: "The bucket read's n and max_temp come off counters that every write maintains." + converged: + why: "Readings are small append-only rows in an hourly INTERVAL-partitioned table, and the rollup is a GROUP BY that prunes to the hours asked for. The last-hour dashboard now scans rows instead of fetching one document, about twice the read bytes in the deck's model." + look: "The insert reports 3 rows affected, and the GROUP BY returns n, average and max per machine and hour." + mongo: + why: "The anomaly team keeps db.aggregate() through the MongoDB API and runs the rollup as one $sql stage (Oracle's addition to the aggregation pipeline: a full SQL statement as one stage) over the same partitioned rows, planned by the optimizer." + look: "One rollup document per machine and hour, with n, avg and max matching the SQL GROUP BY." + measure: + why: "The ratio is the bucket's redo for 3 appends divided by the redo for 3 row inserts. Each append rewrites the bucket as it stands, so the ratio grows with the readings in it; the inserts cost the same whether the hour holds 3 readings or 3,600." + look: "Your dot at 3 readings, near 5×, on a reference line that reaches about 470× at 3,600 readings (one a second for an hour)." +measure: + x_label: "readings in the bucket after the write" + lab_x: 3 + deck_slides: "22–25" + calibration: + - { x: 3, ratio: 4.89 } # doc 4776 B, conv 976 B + - { x: 100, ratio: 24.57 } # doc 24664 B, conv 1004 B + - { x: 1000, ratio: 109.18 } # doc 109616 B, conv 1004 B + - { x: 3600, ratio: 469.84 } # doc 471716 B, conv 1004 B +--- +# Pattern 03: Bucket (time-series) + +**Manufacturing / IoT.** Ingest millions of tiny sensor readings; read them back as +per-machine, per-hour summaries. + +> The textbook write-amplification pattern. Let the engine amortize, not your schema. + +## The document bet + +The platform (all numbers illustrative): 18,000 sensors across 40 plants. Most report +once a second; the hot ones (spindles, vibration) report ten times a second. That is +about 34,000 readings a second, three billion a day. + +A document per reading means billions of tiny documents. The Bucket pattern groups +them: one bucket document per sensor per hour, each reading pushed into an array, +running count, sum, min and max kept on top. Fewer, fatter documents, and a live +dashboard that reads the whole hour as one contiguous document with the counters +already computed. The read side is genuinely good. + +| Access pattern | Who | Rate | Type | +|---|---|---|---| +| Ingest a reading | gateways | ~34k/s | write, hot document | +| Last-hour machine dashboard | line supervisors | 6/min | read, 1 bucket | +| Shift rollup (8 h) | plant managers | 120/day | read, range | +| 7-day anomaly scan, all sensors | reliability engineers | hourly | read, times out | +| ML feature extraction | data science | nightly | read, scan | + +## Immutable facts, mutable container + +In the canonical form (the logical model of plants, machines and readings) a reading +is an immutable fact: a value at a timestamp. In the relational projection it is one +row, inserted once and never changed. The hourly count, sum, min and max are not in +the canonical form at all; they are *derived*, and can be computed with a `GROUP BY` on +read or kept in a materialized view. + +The bucket takes those immutable facts and wraps them in the one mutable thing in the +system. One hot sensor-hour at ten readings a second is 36,000 rewrites of a bucket that +averages about 2.2 MB: roughly 78 GB of bytes moved to store 4.3 MB of data. The same +hour as rows is 36,000 small inserts, about 9 MB, appended to one hourly partition. +Engines soften the constant with in-memory deltas and checkpoints; they don't change the +shape. The write amplification comes from the wrapper, not the data. + +## Where the needle flips + +Diversity is medium (four read shapes) and the workload is write-heavy, but the knob +that bites is **update locality**. Each reading is an array `$push` **plus** a counter +update, a read-modify-write that **rewrites the entire growing bucket document on every +single append.** The bytes moved per hour grow with the square of the readings, so the +redo each append costs climbs all hour. You built a structure whose write cost +increases the more you use it. + +The deck's illustrative cost model (not a benchmark) prices both sides honestly: 120 B +per reading; each `$push` rewrites the whole bucket; a row insert adds about 130 B of +header, index and redo; and the live dashboard reads the hour 360 times an hour, with +the row range scan paying about 150 B more per row than one document fetch. The +`GROUP BY` is not free. + +- **Below about 900 readings a bucket, the bucket wins.** A sensor that reports once a + minute holds 60 readings an hour and is read hundreds of times. Bucket it: the + one-document read beats the rewrites. +- **At one reading a second** (3,600 a bucket) the bucket already costs about 2.7× the + rows. +- **The hot spindle at 36,000** costs about 22×, and its bucket marches at the + **16 MB document ceiling** (about 140,000 readings at this size), where writes simply + start to fail. + +The bucket is a low-rate pattern. It loses exactly on the sensors you care most about: +the fast, hot ones. + +## The converged softening + +Stop hand-rolling buckets for the hot sensors. Let the partition be the bucket: **range +partitioning** does the amortization the bucket was faking, and `GROUP BY` does the +rollup: + +- Readings are tiny, append-only rows in an **INTERVAL-partitioned** table: Oracle + opens a new hourly partition on its own as time advances. Each reading is one small + row: no array to grow, no document to rewrite, no 16 MB ceiling. +- The per-hour summary is a `GROUP BY` over a partition range; asking for one hour + **prunes** to one partition. Where reads repeat (shift reports), a materialized view + precomputes it. +- The anomaly team keeps its aggregation pipeline. `$sql` is Oracle's addition to the + MongoDB aggregation pipeline: one more stage of `db.aggregate()` whose body is a full + SQL statement, with rows returned to the driver as documents. The rollup inside it + gets the cost-based optimizer, parallel execution and pruning to the partitions it + needs, with none of the pipeline's 100 MB stage-memory or 16 MB output caps (see the + root README). + +**The price.** The last-hour dashboard now scans rows instead of fetching one document: +about twice the read bytes in the deck's model. What that buys is a constant-cost insert +with no 16 MB ceiling. An anonymized field result for exactly this reshape: a 29-second +bucket rollup fell to **under 400 milliseconds**, paid for with that heavier last-hour +read. + +**When to flip back: slow sensors.** At one reading a minute a bucket holds 60 readings, +well below the ~900 break-even, and the one-document bucket read is still the better +shape. + +## In this folder + +| File | What it shows | +|---|---| +| `01-document-model.sql` | Per-hour bucket doc; each reading `$push` + counter rewrites the growing bucket | +| `02-converged.sql` | INTERVAL-partitioned readings (constant-cost inserts) + `GROUP BY` rollup + summary MV | +| `01-document-model.js` | The `$push` bucket through the MongoDB API: the same write amplification, runnable in `mongosh` | +| `02-sql-in-pipeline.js` | The hourly rollup as **full SQL through the MongoDB API** via a `$sql` pipeline stage | +| `03-parity.js` | Asserts the `$sql` rollup == the SQL `GROUP BY` == the maintained bucket counters | +| `_capture.sql` | Helper: emits the SQL-lane rollup for the parity check | + +## Validated + +Both lanes run clean on **Oracle AI Database 26ai Free (`23.26.3-faststart`)**. The +`$sql`-in-pipeline rollup returns through the MongoDB API, and the parity assertion +confirms it equals the SQL `GROUP BY` (same machine/hour COUNT/AVG/MAX). +Run with `../../run.sh 03-bucket`. diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/_capture.sql b/labs/converged-modeling-patterns/patterns/03-bucket/_capture.sql new file mode 100644 index 0000000..a8f3a77 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/_capture.sql @@ -0,0 +1,17 @@ +-- Capture the SQL-lane hourly rollup (the 02-converged.sql GROUP BY) as ONE JSON +-- array string. run.sh runs this via sqlplus and injects the result into the +-- parity .js as $SQL_RESULT, which compares it to the Mongo $sql rollup. +SET HEADING OFF PAGESIZE 0 FEEDBACK OFF VERIFY OFF TRIMSPOOL ON +SET LINESIZE 32767 LONG 100000000 LONGCHUNKSIZE 1000000 +SELECT JSON_ARRAYAGG( + JSON_OBJECT('machineId' VALUE machine_id, + 'metric' VALUE metric, + 'hourStart' VALUE TO_CHAR(hs,'YYYY-MM-DD"T"HH24'), + 'n' VALUE n, + 'avg' VALUE avg_val, + 'max' VALUE max_val) + ORDER BY machine_id, metric, hs RETURNING CLOB) +FROM ( SELECT machine_id, metric, TRUNC(reading_ts,'HH24') hs, + COUNT(*) n, ROUND(AVG(reading_val),3) avg_val, MAX(reading_val) max_val + FROM bk_sensor_readings + GROUP BY machine_id, metric, TRUNC(reading_ts,'HH24') ); diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/calibrate.sql b/labs/converged-modeling-patterns/patterns/03-bucket/calibrate.sql new file mode 100644 index 0000000..fd67494 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/calibrate.sql @@ -0,0 +1,26 @@ +-- Calibration resize for Measure it. x = readings in the M-100 / TEMP / 10:00 bucket AFTER +-- the measured write (the lab bucket starts empty and the measured write adds 3, so lab x = 3). +-- Both sides are pre-loaded with :n - 3 readings for that machine-hour (none when :n = 3). +-- Document model: rebuild the bucket with :n - 3 readings and matching counters. +DELETE FROM bk_sensor_doc WHERE JSON_VALUE(data, '$._id') = 'M-100|TEMP|2026-08-01T10'; +INSERT INTO bk_sensor_doc +SELECT JSON_OBJECT('_id' VALUE 'M-100|TEMP|2026-08-01T10', 'machineId' VALUE 'M-100', 'metric' VALUE 'TEMP', + 'hourStart' VALUE '2026-08-01T10:00:00Z', + 'count' VALUE cnt, 'sum' VALUE NVL(total, 0), 'max' VALUE mx, + 'readings' VALUE COALESCE(readings, JSON_ARRAY(RETURNING JSON)) + RETURNING JSON) +FROM (SELECT COUNT(*) AS cnt, SUM(val) AS total, MAX(val) AS mx, + JSON_ARRAYAGG(JSON_OBJECT('ts' VALUE ts, 'val' VALUE val) ORDER BY k RETURNING JSON) AS readings + FROM (SELECT k, 80 + MOD(k * 7, 150) / 10 AS val, + TO_CHAR(TIMESTAMP '2026-08-01 10:00:00' + NUMTODSINTERVAL(k, 'SECOND'), 'YYYY-MM-DD"T"HH24:MI:SS"Z"') AS ts + FROM (SELECT LEVEL AS k FROM dual CONNECT BY LEVEL <= :n) + WHERE k <= :n - 3)); +-- Converged: the same :n - 3 readings as rows in that hour's partition. +DELETE FROM bk_sensor_readings +WHERE machine_id = 'M-100' AND metric = 'TEMP' +AND reading_ts >= TIMESTAMP '2026-08-01 10:00:00' AND reading_ts < TIMESTAMP '2026-08-01 11:00:00'; +INSERT INTO bk_sensor_readings (machine_id, metric, reading_val, reading_ts) +SELECT 'M-100', 'TEMP', 80 + MOD(k * 7, 150) / 10, TIMESTAMP '2026-08-01 10:00:00' + NUMTODSINTERVAL(k, 'SECOND') +FROM (SELECT LEVEL AS k FROM dual CONNECT BY LEVEL <= :n) +WHERE k <= :n - 3; +COMMIT; diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/figures/doc-shape.svg b/labs/converged-modeling-patterns/patterns/03-bucket/figures/doc-shape.svg new file mode 100644 index 0000000..79acb65 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/figures/doc-shape.svg @@ -0,0 +1 @@ +readings · one bucket per sensor per hour{"_id": "M-118|VIB|2026-09-25T14","machineId": "M-118","metric": "VIB","hourStart": "2026-09-25T14:00:00Z","count": 21604,"sum": 1548212.7,"min": 0.8, "max": 97.3,"readings": [{ "ts": "14:00:00.1", "val": 71.4 },{ "ts": "14:00:00.2", "val": 71.9 },... 21,601 more readings ...{ "ts": "14:36:00.4", "val": 72.2 }]}counters: $inc / $maxrewritten on everysingle reading$push lands here:rewrites 2.6 MBand it only grows \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/figures/erd.svg b/labs/converged-modeling-patterns/patterns/03-bucket/figures/erd.svg new file mode 100644 index 0000000..f7386c0 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/figures/erd.svg @@ -0,0 +1 @@ +CANONICAL FORM: EACH READING ONCE, NEVER REWRITTENONE HOT SENSOR-HOUR, BUCKETEDPLANTPKplant_idvarchar2namevarchar2regionvarchar2MACHINEPKmachine_idvarchar2FKplant_idvarchar2modelvarchar2line_nonumberSENSOR_READINGSPKreading_idnumberFKmachine_idvarchar2metricvarchar2reading_valnumberreading_tstimestampHOURLY_MV · derivedmachine_idvarchar2hour_startdatennumbersum_valnumbermax_valnumberGROUP BY hour · on readM-118 · spindle vibration10 readings / s1 bucket documentcount·sum·min·maxreadings[ ]every reading rewrites bothO(n²) rewrite volume36,000 rewrites · 4.3 MB bucket≈ 78 GB if each push rewrites itRelational projection36,000 row inserts · ≈ 9 MBappended to one partitionKey:red: rewritten on every reading (the bucket's counters and array)green: immutable (SENSOR_READINGS rows, inserted once) \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/figures/flow.svg b/labs/converged-modeling-patterns/patterns/03-bucket/figures/flow.svg new file mode 100644 index 0000000..a5a5903 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/figures/flow.svg @@ -0,0 +1 @@ +RELATIONAL PROJECTIONSUMMARYCONSUMERSmachinesplant · line · modelsensor_readingsappend-only rowsINTERVAL 1 hGROUP BY hourpartition-prunedparallelhourly_mvsummary MVDashboardsSQL · last hourAnomaly scanMongo API $sqlShift reportSQL · MV \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/03-bucket/figures/model-curve.svg b/labs/converged-modeling-patterns/patterns/03-bucket/figures/model-curve.svg new file mode 100644 index 0000000..90af688 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/03-bucket/figures/model-curve.svg @@ -0,0 +1 @@ +KNOB SETTINGS · THIS WORKLOAD1 · DiversityMedium: 4 read shapesdashboard, shift rollup,anomaly scan, ML features2 · Read / writeWrite-heavy~3B readings in per day;reads are rollups, not points3 · Update localityEvery write, same documenteach reading rewrites thegrowing hour bucketBYTES PER SENSOR-HOUR vs READINGS / BUCKET · illustrative modelbucket is cheaperrows are cheaper100 KB10 MB1 GB100 GB1001k10k100kreadings per bucket (log)bytes touched per sensor-hour (log)break-even ≈ 9001 / min · 601 / s · 3,600hot · 36,000Bucket · $push + countersRows · append, GROUP BY on read16 MB cap ≈ 140k \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/04-subset/01-document-model.sql b/labs/converged-modeling-patterns/patterns/04-subset/01-document-model.sql new file mode 100644 index 0000000..ef65b37 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/04-subset/01-document-model.sql @@ -0,0 +1,71 @@ +-- ============================================================================ +-- Pattern 04 · Subset (hot inline / vertical partition) · THE DOCUMENT MODEL +-- Industry: insurance. Access pattern: the policy page shows the 3 most recent +-- claims instantly; the full claim history is opened rarely. +-- +-- The document move: keep the hot subset (recent N claims) INLINE on the policy +-- doc so the common read touches one document, and push the cold tail into a +-- separate history collection. Keep the working set in RAM. +-- +-- THE WRITE-AMPLIFICATION COST: to keep the inline subset "recent," EVERY new +-- claim pays twice: a push-and-trim rewrite of the WHOLE policy document, AND an +-- insert of the full claim into the history collection. Two writes, two places, +-- for one fact. The bet is that the summary read dwarfs the claim write, and on a +-- normal day it does (deck: ~2M summary reads vs ~30k events, break-even ~83k +-- events/day). On a catastrophe day (~300k events) the bet loses: the subset +-- costs ~40% more than querying, all of it on the hottest policy documents. +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @01-document-model.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables + WHERE table_name IN ('SB_POLICY_DOC','SB_CLAIM_HISTORY') + ORDER BY table_name) LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' PURGE'; + END LOOP; +END; +/ + +-- sb_policy_doc = policy + inline recent claims (hot); sb_claim_history = cold history. +CREATE JSON COLLECTION TABLE sb_policy_doc; +CREATE JSON COLLECTION TABLE sb_claim_history; + +-- Policy P-001 with 3 recent claims inline (oldest -> newest). +INSERT INTO sb_policy_doc VALUES (JSON('{"_id":"P-001","holder":"Dana Holder","product":"AUTO", + "recentClaims":[ + {"claimId":"CL-1001","amount":800}, + {"claimId":"CL-1002","amount":1500}, + {"claimId":"CL-1003","amount":2300}]}')); +COMMIT; + +-- A new claim CL-1004 arrives. To keep the inline subset at N=3 you must: +-- (a) append it to the inline array, (b) trim the oldest, (c) write the full +-- claim to the history collection. Two writes, two places, for one claim. +-- (a) append newest + (b) trim oldest: one statement that rewrites the whole policy document. +-- @step Push CL-1004 in, trim the oldest claim +-- @note Append plus trim rewrites the WHOLE policy document: the first of two writes for a single new claim. +-- @why Update locality is the hot knob: every claim event lands on the parent policy, and keeping the list recent is a read-modify-write of the whole policy document (append, trim, rewrite) on every event. +-- @look Rows affected is 1; Measure it shows the redo of rewriting the policy next to one claim insert. +-- @figure doc-shape.svg The policy document as built: recent claim events inline, every event also written to a history collection +-- @measure claim-event +UPDATE sb_policy_doc +SET data = JSON_TRANSFORM(data, + APPEND '$.recentClaims' = JSON_OBJECT('claimId' VALUE 'CL-1004', 'amount' VALUE 4200), + REMOVE '$.recentClaims[0]') +WHERE JSON_VALUE(data,'$._id') = 'P-001'; +-- (c) the full claim also goes to the history collection: the second write. +-- @step Write the full claim to the history collection +-- @note The second write for the same claim: the history collection must be kept in step with the inline subset by hand. +-- @why Update locality again: the same fact lands in a second place, and if this write fails after the first, the summary and the history disagree. +-- @look Rows affected is 1, the second write for a single claim. +INSERT INTO sb_claim_history VALUES (JSON('{"_id":"CL-1004","policyId":"P-001","amount":4200}')); +COMMIT; + +-- The common read the pattern optimizes for: recent claims, one document. +-- @step Read the recent claims (one document) +-- @note recentClaims now shows CL-1004 in, CL-1001 trimmed out: two writes paid so this read is one document fetch. +-- @why Read/write is the subset's case: at about 67 summary reads per claim event, the policy page reads one document with the list already built. +-- @look recent_claims_inline holds 3 claims, the list every event paid to keep sorted and trimmed. +SELECT JSON_VALUE(data,'$._id') AS policy, + JSON_QUERY(data,'$.recentClaims') AS recent_claims_inline +FROM sb_policy_doc WHERE JSON_VALUE(data,'$._id') = 'P-001'; diff --git a/labs/converged-modeling-patterns/patterns/04-subset/02-converged.sql b/labs/converged-modeling-patterns/patterns/04-subset/02-converged.sql new file mode 100644 index 0000000..3b4c90a --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/04-subset/02-converged.sql @@ -0,0 +1,98 @@ +-- ============================================================================ +-- Pattern 04 · Subset (hot inline / vertical partition) · THE CONVERGED ALTERNATIVE +-- "Stop maintaining the subset. Query it." +-- +-- The canonical form is policy, claim and claim event; a claim event is an +-- immutable fact recorded once. Here it is projected as rows on ONE claims +-- table: hot and cold live together. A composite index on +-- (policy_id, claim_ts DESC) makes "the recent N" an index range scan that stops +-- after N rows: no maintenance rewrite, no history collection, no trimming. Every +-- claim is one insert plus one index entry; the "recent" slice is derived on +-- read, not maintained on write. SQL/JSON assembles the same policy document +-- (policy header plus its recent claims) at read time, so hot and cold are +-- reunited by the query, never by a maintenance job. +-- +-- The tradeoff: each write drops from a document rewrite plus a history insert +-- to one plain insert, and the rarer full-history read is the same query without +-- the FETCH FIRST. The price is the read: the summary now does more work than a +-- single document fetch (~25% in the deck's model). Below ~83k events/day the +-- subset is cheaper; a 300k-event catastrophe day is where querying wins. If the +-- inline slice is frozen at write time (the coverages on the declarations page), +-- keep it in the document. The subset stops being state you maintain and becomes +-- a projected shape you ask for. +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @02-converged.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables + WHERE table_name IN ('SB_CLAIMS','SB_POLICIES') + ORDER BY table_name) LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' CASCADE CONSTRAINTS PURGE'; + END LOOP; +END; +/ + +CREATE TABLE sb_policies ( + policy_id VARCHAR2(12) PRIMARY KEY, + holder VARCHAR2(120) NOT NULL, + product VARCHAR2(16) NOT NULL +); + +CREATE TABLE sb_claims ( + claim_id VARCHAR2(12) PRIMARY KEY, + policy_id VARCHAR2(12) NOT NULL REFERENCES sb_policies(policy_id), + amount NUMBER(14,2) NOT NULL, + claim_ts TIMESTAMP DEFAULT SYSTIMESTAMP NOT NULL +); + +-- The index that turns "recent N" into a range scan that stops after N rows. +CREATE INDEX sb_ix_claims_recent ON sb_claims (policy_id, claim_ts DESC); + +INSERT INTO sb_policies VALUES ('P-001','Dana Holder','AUTO'); +INSERT INTO sb_claims VALUES ('CL-1001','P-001', 800, TIMESTAMP '2026-01-10 09:00:00'); +INSERT INTO sb_claims VALUES ('CL-1002','P-001',1500, TIMESTAMP '2026-03-22 09:00:00'); +INSERT INTO sb_claims VALUES ('CL-1003','P-001',2300, TIMESTAMP '2026-05-30 09:00:00'); +-- CL-1004 is just one plain insert -- no trim, no history write, no doc rewrite. +-- @step Insert claim CL-1004 +-- @note One plain insert: no trim, no history write, no policy document to rewrite. +-- @why Update locality stops concentrating: the claim is one row and one index entry, with no policy document to rewrite and no history to keep in step. On a CAT day that is 300,000 inserts, not 300,000 policy rewrites. +-- @look Rows affected is 1; Measure it shows redo near 1 KB whatever the size of the inline list. +-- @figure erd.svg Policy, claim and claim event in canonical form, beside how one claim event lands in the subset model +-- @measure claim-event +INSERT INTO sb_claims VALUES ('CL-1004','P-001',4200, TIMESTAMP '2026-08-01 09:00:00'); +COMMIT; + +-- The hot read: recent 3 claims = index range scan on sb_ix_claims_recent + FETCH +-- FIRST. Nothing was maintained to make this fast; the read pays for a short scan instead. (EXPLAIN PLAN shows the range +-- scan stops after 3 rows -- no sort of the full history.) +-- @step Read the recent 3 claims (index range scan) +-- @note CL-1004 is already the top row: nothing was maintained to make this fast, the index just stops after 3. +-- @why Read/write, priced honestly: the recent list is an index range scan that stops after 3 rows, more work than one document fetch, about 25% more per summary read in the deck's model. +-- @look The 3 newest claims in claim_ts order, with nothing stored to keep them current. +SELECT claim_id, amount, claim_ts +FROM sb_claims +WHERE policy_id = 'P-001' +ORDER BY claim_ts DESC +FETCH FIRST 3 ROWS ONLY; + +-- The full policy document -- hot + cold reunited AT READ TIME by SQL/JSON. No +-- inline subset stored, no history collection to keep in step. +-- @step Assemble the full policy document at read time +-- @note Header plus recent claims, reunited by the query: no inline subset stored, no history collection to keep in step. +-- @why Diversity: agents get the policy document they had, assembled by SQL/JSON, and adjusters get the full history from the same rows by dropping FETCH FIRST. +-- @look policy_document shows the header with recentClaims newest first, built by the query rather than stored. +SELECT JSON_OBJECT( + 'policyId' VALUE p.policy_id, + 'holder' VALUE p.holder, + 'product' VALUE p.product, + 'recentClaims' VALUE ( + SELECT JSON_ARRAYAGG( + JSON_OBJECT('claimId' VALUE claim_id, 'amount' VALUE amount, + 'ts' VALUE TO_CHAR(claim_ts,'YYYY-MM-DD')) + ORDER BY claim_ts DESC RETURNING CLOB) + FROM ( SELECT claim_id, amount, claim_ts FROM sb_claims + WHERE policy_id = p.policy_id + ORDER BY claim_ts DESC FETCH FIRST 3 ROWS ONLY )) + RETURNING CLOB) AS policy_document +FROM sb_policies p +WHERE p.policy_id = 'P-001'; diff --git a/labs/converged-modeling-patterns/patterns/04-subset/README.md b/labs/converged-modeling-patterns/patterns/04-subset/README.md new file mode 100644 index 0000000..9bd77d5 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/04-subset/README.md @@ -0,0 +1,101 @@ +--- +title: Subset +industry: Insurance +deck: "26–29" +problem: >- + The policy document keeps the ten most recent claim events inline. Every event becomes a + push-sort-trim rewrite of the policy to maintain a list a query could simply read. +knobs: + - { name: Diversity, setting: "High: 4 consumers", help: "Agents, the customer app, adjusters and fraud all read claim events: the first two want the recent few, the last two want everything." } + - { name: Read / write, setting: "Read-heavy, ~67 : 1", help: "About 2 million summary reads a day against about 30,000 claim events, which is the case for keeping the subset inline." } + - { name: Update locality, setting: "Every event hits the parent", hot: true, help: "Every claim event rewrites the parent policy document and writes history too: break-even is about 83,000 events a day, and a 300,000-event CAT day makes the subset about 40% more expensive than querying." } +help: + tabs: + document: + why: "The starting point: keep the most recent claims inline on the policy so the summary page reads one document, and push each full claim into a history collection. Every new claim is a whole-policy rewrite plus a second write that has to stay in step." + look: "recent_claims_inline holds exactly 3 claims, trimmed by hand on every write." + converged: + why: "One claims table and a (policy_id, claim_ts DESC) index make the recent N a range scan that stops after N rows. The summary read does more work than one document fetch, about 25% in the deck's model; what that buys is one insert per claim and nothing to keep in step." + look: "The insert reports 1 row affected, and the recent-claims query returns the newest 3 with no stored subset." + measure: + why: "The ratio is the redo for pushing and trimming the policy document divided by the redo for one claim insert. The document side rewrites every claim kept inline, so the ratio grows with the subset's size; the insert stays near 1 KB." + look: "Your dot at 3 inline claims sits near 1.6×, and it leaves out the history insert the document model also pays." +measure: + x_label: "claims kept inline in the policy document" + lab_x: 3 + deck_slides: "26–29" + calibration: + - { x: 3, ratio: 1.62 } # doc 1680 B, conv 1036 B + - { x: 10, ratio: 1.95 } # doc 2024 B, conv 1036 B + - { x: 100, ratio: 6.12 } # doc 6336 B, conv 1036 B + - { x: 1000, ratio: 26.26 } # doc 26576 B, conv 1012 B +--- +# Pattern 04: Subset (hot inline / vertical partition) + +**Insurance.** The policy summary page shows the most recent claim events instantly +(the deck uses 10; this lab keeps 3 so the output stays readable); the full claim +history is opened far less often. + +> Stop maintaining the subset. Query it. + +## The document bet + +The Subset pattern keeps the hot slice (the recent N claims) **inline** on the +policy document so the common read touches one document, and pushes the cold tail +into a separate history collection. Keep the working set in RAM; keep the hot read +to a single fetch. In the deck's workload (8 million policies, about 2 million +summary reads a day against about 30,000 claim events) that bet is sound: the +summary read dwarfs the event write. + +## Where the needle flips + +To keep the inline subset "recent," **every new claim event** pays twice: a +push-sort-trim **rewrite of the whole policy document**, *and* an insert of the full +event into the history collection. That is two writes in two places for one fact, and +if the second write fails, the summary and the history disagree. It also +concentrates: every event on every claim lands on the same policy document, which +becomes the hottest document in the system because of a list it has to maintain. + +On a normal day the Subset pattern still wins. In the deck's illustrative cost model +(not a benchmark), querying adds about 25% to each summary read, and on 2 million +reads that premium is real. Break-even is about **83,000 events a day**. A +catastrophe (CAT) day, such as a hurricane landfall, runs at about **300,000** +events, and then the Subset pattern costs about **40% more** than querying, with +every extra write a whole-document rewrite on exactly the policies the customer app +is hammering. Size the model for the storm, not the sunny day. + +## The converged softening + +The canonical form is policy, claim and claim event, where a claim event is an +immutable fact recorded once. Here it is projected as rows on **one claims table**: +hot and cold live together. A composite index on `(policy_id, claim_ts DESC)` keeps +each policy's claims pre-sorted, so "the recent N" is an index range scan that +**stops after N rows**. Every claim is one insert and one index entry: no +maintenance rewrite, no history collection, no trimming. SQL/JSON assembles the same +policy document (header plus its recent claims) **at read time**, so hot and cold are +reunited by the query, never by a maintenance job. + +That is the tradeoff, priced honestly: the summary read now does more work than a +single document fetch (about 25% in the deck's model). What it buys is a write path +that does not collapse in a storm, and one copy of every event for fraud review and +audit. The subset stops being *state you maintain* and becomes *a projected shape you +ask for.* The rarer full-history read is the same query without the `FETCH FIRST`. + +**When the document shape is still right:** if the inline slice is frozen at write +time and rarely changes (for example, the three coverages printed on the +declarations page), keep it in the document. A slice that changes on every event is +a query. + +## In this folder + +| File | What it shows | +|---|---| +| `01-document-model.sql` | Recent-N inline + history collection; each claim is a push-and-trim rewrite of the policy plus a history insert | +| `02-converged.sql` | One claims table + descending composite index; recent-N via `FETCH FIRST`; policy document assembled on read | + +## Validated + +Both scripts run clean on **Oracle AI Database 26ai Free (`23.26.3-faststart`)**. This +pattern stays on the SQL and SQL/JSON access surfaces: a claims table has no natural +single collection to read through the MongoDB API, so there is no MongoDB API lane here. +Run them with `../../run.sh 04-subset`. diff --git a/labs/converged-modeling-patterns/patterns/04-subset/calibrate.sql b/labs/converged-modeling-patterns/patterns/04-subset/calibrate.sql new file mode 100644 index 0000000..efed045 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/04-subset/calibrate.sql @@ -0,0 +1,16 @@ +-- Calibration resize for Measure it: policy P-001 keeps :n claims inline on the document +-- side; the converged side holds the same :n claims as rows. Generated claim ids +-- (CL-G000001...) never collide with the measured CL-1004. +DELETE FROM sb_policy_doc WHERE JSON_VALUE(data, '$._id') = 'P-001'; +INSERT INTO sb_policy_doc +SELECT JSON_OBJECT('_id' VALUE 'P-001', 'holder' VALUE 'Dana Holder', 'product' VALUE 'AUTO', + 'recentClaims' VALUE claims RETURNING JSON) +FROM (SELECT JSON_ARRAYAGG(JSON_OBJECT('claimId' VALUE 'CL-G' || LPAD(LEVEL, 6, '0'), 'amount' VALUE 500 + MOD(LEVEL * 173, 4000)) + ORDER BY LEVEL RETURNING JSON) AS claims + FROM dual CONNECT BY LEVEL <= :n); +DELETE FROM sb_claims WHERE policy_id = 'P-001'; +INSERT INTO sb_claims (claim_id, policy_id, amount, claim_ts) +SELECT 'CL-G' || LPAD(LEVEL, 6, '0'), 'P-001', 500 + MOD(LEVEL * 173, 4000), + TIMESTAMP '2026-01-01 00:00:00' + NUMTODSINTERVAL(LEVEL, 'MINUTE') +FROM dual CONNECT BY LEVEL <= :n; +COMMIT; diff --git a/labs/converged-modeling-patterns/patterns/04-subset/figures/doc-shape.svg b/labs/converged-modeling-patterns/patterns/04-subset/figures/doc-shape.svg new file mode 100644 index 0000000..845fb14 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/04-subset/figures/doc-shape.svg @@ -0,0 +1 @@ +policies · one document per policy{"_id": "POL-7730142","holder": "R. Alvarez","product": "HOME","coverages": [ ... ],"recentEvents": [{ "claim": "CLM-5581", "type": "PAYMENT","amount": 2400, "ts": "09-27T14:02" },{ "claim": "CLM-5581", "type": "NOTE" },... 8 more · sorted · trimmed to 10]}rewritten on every event$push · $sort · $slice: 10~30k times a dayclaim_history · one document per event{ "_id": "EV-90311", "policy": "POL-7730142","claim": "CLM-5581", "type": "PAYMENT" }…and a second writesame event, another placeto keep in step \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/04-subset/figures/erd.svg b/labs/converged-modeling-patterns/patterns/04-subset/figures/erd.svg new file mode 100644 index 0000000..be23c6d --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/04-subset/figures/erd.svg @@ -0,0 +1 @@ +CANONICAL FORM: EACH EVENT ONCEONE CLAIM EVENT, TWO MODELSPOLICYPKpolicy_idvarchar2holdervarchar2productvarchar2premiumnumberstatusvarchar2CLAIMPKclaim_idvarchar2FKpolicy_idvarchar2loss_datedatestatusvarchar2reservenumberCLAIM_EVENT · append-onlyPKevent_idvarchar2FKclaim_idvarchar2FKpolicy_idvarchar2event_typevarchar2amountnumberevent_tstimestampINDEXpolicy_id, event_ts DESCclaim event · CLM-5581payment · $2,400policy docpush · sort · trimwhole doc rewrittenhistory docinsert full eventsecond place2 writes, 2 places, per eventthe hot doc absorbs every oneRelational projection1 row insert + 1 index entrythe recent 10 is derived on read,not maintainedmutable claim state: updated in placeimmutable event facts: append once, never rewritten \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/04-subset/figures/flow.svg b/labs/converged-modeling-patterns/patterns/04-subset/figures/flow.svg new file mode 100644 index 0000000..382bcb8 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/04-subset/figures/flow.svg @@ -0,0 +1 @@ +RELATIONAL PROJECTIONSAME ROWS, TWO READSCONSUMERSpoliciesheader, onceclaim_eventsappend · DESC indexclaimsstatus · reservepolicy summarySQL/JSONFETCH FIRST 10full historysame query,no FETCH FIRSTAgents · appSQL/JSON docAdjustersSQL · auditFraud reviewSQL analytics \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/04-subset/figures/model-curve.svg b/labs/converged-modeling-patterns/patterns/04-subset/figures/model-curve.svg new file mode 100644 index 0000000..f9ae95f --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/04-subset/figures/model-curve.svg @@ -0,0 +1 @@ +KNOB SETTINGS · THIS WORKLOAD1 · DiversityHigh: 4 consumersagents, customer app, adjustersand fraud all read claim events2 · Read / writeRead-heavy, ~67 : 12M summary reads/day against~30k claim events/day3 · Update localityEvery event hits the parenteach event rewrites the policydoc and writes history tooDAILY WORK vs CLAIM EVENTS / DAY · illustrative modelthe Subset pattern is cheaperquerying is cheaper3M5M10M10k100k1Mclaim events per day (log)work units per day (log)break-even ≈ 83k / daynormal day · 30kCAT day · 300kSubset · push + trim + historyQuery it · FETCH FIRST 10, +25% read \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/01-document-model.sql b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/01-document-model.sql new file mode 100644 index 0000000..3cc6dd6 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/01-document-model.sql @@ -0,0 +1,75 @@ +-- ============================================================================ +-- Pattern 05 · Tree / Hierarchy (materialized path) · THE DOCUMENT MODEL +-- Industry: manufacturing. Access pattern: explode a bill of materials -- read a +-- whole subassembly's components fast; ask "what is this part used in?" +-- +-- The document move: store each component with a MATERIALIZED PATH -- the chain +-- of ancestors as a string ("/P-1000/P-1100/P-1110"). A subtree read becomes a +-- left-anchored prefix scan (LIKE '/P-1000/P-1100%'): an index range scan, +-- O(log n). The cheapest explosion there is, and that is real: for a tree that +-- rarely moves and is read one way (a catalogue, an org chart snapshot), or +-- moves well under ~1,000 parts, this is still the right shape. +-- +-- THE WRITE-AMPLIFICATION COST: the path encodes position, so moving one +-- subassembly rewrites the path of EVERY descendant. Re-parent a wheelset and +-- you rewrite the wheelset, the wheel, the spoke, every node beneath it -- write +-- amplification proportional to the size of the subtree, for a single logical +-- move. And a genuine graph question -- "which assemblies use this spoke?" +-- (upward, multi-parent, N hops) -- a downward prefix string simply cannot express, +-- so teams bolt on a usedIn parent list the app must keep in sync by hand. +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @01-document-model.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables WHERE table_name = 'TR_BOM_DOC') LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' PURGE'; + END LOOP; +END; +/ + +CREATE JSON COLLECTION TABLE tr_bom_doc; + +-- A bicycle BOM as a tree of materialized paths. +INSERT INTO tr_bom_doc VALUES (JSON('{"_id":"P-1000","name":"Bicycle","path":"/P-1000"}')); +INSERT INTO tr_bom_doc VALUES (JSON('{"_id":"P-1100","name":"Wheelset","path":"/P-1000/P-1100"}')); +INSERT INTO tr_bom_doc VALUES (JSON('{"_id":"P-1110","name":"Wheel","path":"/P-1000/P-1100/P-1110"}')); +INSERT INTO tr_bom_doc VALUES (JSON('{"_id":"P-1111","name":"Spoke","path":"/P-1000/P-1100/P-1110/P-1111"}')); +INSERT INTO tr_bom_doc VALUES (JSON('{"_id":"P-1200","name":"Frame","path":"/P-1000/P-1200"}')); +COMMIT; + +-- The read the pattern optimizes for: everything under the wheelset = one prefix +-- scan. This is the genuine strength of the materialized path. +-- @step Read a subtree with a prefix scan +-- @note Everything under the wheelset in one index range scan: the genuine strength of the materialized path. +-- @why Read/write is the path's case: two million explosions a day each become one left-anchored prefix scan, the cheapest hierarchy read there is. +-- @look Wheelset, wheel and spoke come back in path order from one LIKE prefix predicate. +-- @figure doc-shape.svg One part document as built: a materialized path encodes its position +SELECT JSON_VALUE(data,'$._id') AS part, JSON_VALUE(data,'$.name') AS name +FROM tr_bom_doc +WHERE JSON_VALUE(data,'$.path') LIKE '/P-1000/P-1100%' +ORDER BY JSON_VALUE(data,'$.path'); + +-- --------------------------------------------------------------------------- +-- THE NEEDLE FLIPS HERE. Re-parent the wheelset under the frame. One logical +-- move, but every descendant's path must be rewritten -- a whole-document +-- rewrite per node in the subtree. +-- --------------------------------------------------------------------------- +-- @step Re-parent the wheelset under the frame +-- @note One logical move, but every descendant's path must be rewritten: a whole-document rewrite per node in the subtree. +-- @why Update locality is the hot knob: the path stores position, so one move is a read-modify-write of every part document beneath the moved node. A 40,000-part module move is 40,000 rewrites. +-- @look Rows affected is 3, one per part in the wheelset's subtree; Measure it shows the redo those rewrites cost. +-- @measure reparent +UPDATE tr_bom_doc +SET data = JSON_TRANSFORM(data, + SET '$.path' = REPLACE(JSON_VALUE(data,'$.path'), + '/P-1000/P-1100', '/P-1000/P-1200/P-1100')) +WHERE JSON_VALUE(data,'$.path') LIKE '/P-1000/P-1100%'; +COMMIT; + +-- @step Confirm every descendant moved +-- @note Count of docs now carrying the new path prefix: one logical move, one rewrite per descendant. +-- @why Update locality again: the count is how many documents one engineering change touched. At production scale those rewrites run as a batch, not one atomic change. +-- @look The write_amplification line reports 3 descendant docs rewritten. +SELECT 'reparent rewrote ' || COUNT(*) || ' descendant docs' AS write_amplification +FROM tr_bom_doc +WHERE JSON_VALUE(data,'$.path') LIKE '/P-1000/P-1200/P-1100%'; diff --git a/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/02-converged.sql b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/02-converged.sql new file mode 100644 index 0000000..5f05c47 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/02-converged.sql @@ -0,0 +1,124 @@ +-- ============================================================================ +-- Pattern 05 · Tree / Hierarchy · THE CONVERGED ALTERNATIVE +-- Split the two questions the materialized path was trying to answer at once. +-- In the canonical form (the logical model) each parent -> child link is one +-- relationship; this relational projection stores it as one edge row, and +-- position is derived by walking the edges. +-- +-- TREE structure lives in ADJACENCY EDGES (parent_id -> child_id). Re-parenting +-- is a SINGLE edge update -- the subtree follows automatically because position +-- is not baked into every node. Write amplification for a reorg drops from +-- O(subtree) to O(1). +-- +-- SUBTREE / EXPLOSION reads use native recursion -- CONNECT BY streams the +-- frontier in one pass (emit CONNECT BY or GRAPH_TABLE for hierarchy, never a +-- recursive CTE, which dams each level into an intermediate relation the CBO +-- can barely optimize). The walk is not free: it costs about 30% more per +-- explosion than a prefix scan in the deck's illustrative model. +-- +-- GENUINE GRAPH questions -- "which assemblies use this wheel?", multi-parent, +-- N hops, the DAG a downward prefix string cannot express -- use SQL/PGQ +-- GRAPH_TABLE over the SAME rows. No separate graph database to sync. +-- +-- The needle-flip, resolved: the write that the path made expensive (reorg) +-- becomes one row, and the read the path could not do at all (where-used across +-- shared components) becomes one graph match. +-- +-- THE TRADEOFF: break-even sits near 1,000 parts per move. Below that, or for a +-- static tree read one way, the path's prefix scan is cheaper; a 10,000-part +-- move makes the path's day ~3x the edge model's, ~10x at 40,000. +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @02-converged.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables + WHERE table_name IN ('TR_BOM_EDGES','TR_PARTS') + ORDER BY table_name) LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' CASCADE CONSTRAINTS PURGE'; + END LOOP; +END; +/ + +CREATE TABLE tr_parts ( + part_id VARCHAR2(12) PRIMARY KEY, + name VARCHAR2(60) NOT NULL +); + +-- The tree/DAG is EDGES, not baked paths. One row per parent->child link. +CREATE TABLE tr_bom_edges ( + edge_id NUMBER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + parent_id VARCHAR2(12) NOT NULL REFERENCES tr_parts(part_id), + child_id VARCHAR2(12) NOT NULL REFERENCES tr_parts(part_id), + qty NUMBER(10) DEFAULT 1 NOT NULL +); +CREATE INDEX tr_ix_edge_parent ON tr_bom_edges (parent_id); +CREATE INDEX tr_ix_edge_child ON tr_bom_edges (child_id); + +INSERT INTO tr_parts VALUES ('P-1000','Bicycle'); +INSERT INTO tr_parts VALUES ('P-1100','Wheelset'); +INSERT INTO tr_parts VALUES ('P-1110','Wheel'); +INSERT INTO tr_parts VALUES ('P-1111','Spoke'); +INSERT INTO tr_parts VALUES ('P-1200','Frame'); +-- P-1300 (Trailer) shares the wheel with the bicycle -- a DAG, not a pure tree. +INSERT INTO tr_parts VALUES ('P-1300','Trailer'); +INSERT INTO tr_bom_edges (parent_id,child_id,qty) VALUES ('P-1000','P-1100',1); +INSERT INTO tr_bom_edges (parent_id,child_id,qty) VALUES ('P-1000','P-1200',1); +INSERT INTO tr_bom_edges (parent_id,child_id,qty) VALUES ('P-1100','P-1110',2); +INSERT INTO tr_bom_edges (parent_id,child_id,qty) VALUES ('P-1110','P-1111',32); +INSERT INTO tr_bom_edges (parent_id,child_id,qty) VALUES ('P-1300','P-1110',2); +COMMIT; + +CREATE OR REPLACE PROPERTY GRAPH tr_bom_graph + VERTEX TABLES ( tr_parts KEY (part_id) LABEL part PROPERTIES (part_id, name) ) + EDGE TABLES ( tr_bom_edges KEY (edge_id) + SOURCE KEY (parent_id) REFERENCES tr_parts (part_id) + DESTINATION KEY (child_id) REFERENCES tr_parts (part_id) + LABEL contains PROPERTIES (qty) ); + +-- READ 1 -- explode the bicycle (all components, any depth): CONNECT BY, one pass. +-- @step Explode the bicycle with CONNECT BY +-- @note Native recursion streams the whole frontier in one pass: no recursive CTE damming each level into an intermediate relation. +-- @why Read/write, priced honestly: CONNECT BY walks the edges from the root down in one statement, about 30% more work per explosion than a prefix scan in the deck's model. +-- @look Each component with its qty and depth, indented under its parent. +-- @figure erd.svg Part, BOM edge and change order in canonical form, beside the fan-out of one re-parent across materialized paths +SELECT LPAD(' ', 2*(LEVEL-1)) || child_id AS component, qty, LEVEL AS depth +FROM tr_bom_edges +START WITH parent_id = 'P-1000' +CONNECT BY PRIOR child_id = parent_id +ORDER SIBLINGS BY child_id; + +-- READ 2 -- where-used: which assemblies contain the wheel? The multi-parent, +-- N-hop question the materialized path could not express. SQL/PGQ, same rows. +-- @step Ask a graph question: where is the wheel used? +-- @note Multi-parent, N-hop, upward traversal: a downward prefix string simply cannot express this. +-- @why Diversity: where-used runs upward across shared parts, which a downward path cannot express. A SQL/PGQ graph over the same two tables answers it with no second structure to maintain. +-- @look P-1300, the trailer that shares the wheel, appears alongside the bicycle's assemblies. +SELECT assembly +FROM GRAPH_TABLE (tr_bom_graph + MATCH (a IS part)-[IS contains]->{1,4}(p IS part) + WHERE p.part_id = 'P-1110' + COLUMNS (a.part_id AS assembly)); +-- ^ returns P-1100, P-1000, AND P-1300 (the trailer that shares the wheel). + +-- THE NEEDLE-FLIP, RESOLVED -- re-parent the wheelset under the frame: ONE row. +-- The entire subtree moves with it; no descendant paths to rewrite. +-- @step Re-parent the wheelset under the frame +-- @note One edge update: the entire subtree moves with it, no descendant paths to rewrite. +-- @why Update locality collapses: position is derived from the edges, so the move is one edge row however many parts hang beneath it. +-- @look Rows affected is 1; Measure it shows redo near 1 KB for a subtree of any size. +-- @measure reparent +UPDATE tr_bom_edges SET parent_id = 'P-1200' +WHERE parent_id = 'P-1000' AND child_id = 'P-1100'; +COMMIT; + +-- Re-explode: the wheelset (and everything under it) now hangs off the frame, +-- from that single edge change. +-- @step Re-explode: confirm the subtree moved with the edge +-- @note The wheelset (and everything under it) now hangs off the frame, from that single edge change. +-- @why Update locality, resolved: the next explosion walks the edges as they are now, so the whole subtree moved with that one row. +-- @look P-1100 and its children now appear under P-1200. +SELECT LPAD(' ', 2*(LEVEL-1)) || child_id AS component, LEVEL AS depth +FROM tr_bom_edges +START WITH parent_id = 'P-1000' +CONNECT BY PRIOR child_id = parent_id +ORDER SIBLINGS BY child_id; diff --git a/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/README.md b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/README.md new file mode 100644 index 0000000..98507ef --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/README.md @@ -0,0 +1,128 @@ +--- +title: Tree / Hierarchy +industry: Manufacturing bill of materials +deck: "30–33" +problem: >- + Each part carries a materialized path. Re-parenting a subassembly rewrites every + descendant, and "which products use this part?" cannot be answered from a path. +knobs: + - { name: Diversity, setting: "High: 4 questions", help: "Planning explodes an assembly, costing rolls it up, engineering asks where-used and quality asks recall impact: four questions over one structure, two of them upward." } + - { name: Read / write, setting: "Read-heavy", help: "About 2 million explosions a day against about 200 engineering change orders, so on this knob alone the path's prefix scan is the right read." } + - { name: Update locality, setting: "Lands on every descendant", hot: true, help: "One re-parent rewrites the path of every part in the moved subtree, 10 to 40,000 parts: break-even is about 1,000 parts a move, and a 10,000-part move makes the path's day about 3× the edge model's." } +help: + tabs: + document: + why: "The starting point: every part carries a materialized path, so exploding an assembly is one prefix range scan. Position is stored in every node, so moving a subassembly rewrites every descendant, and the path cannot answer where-used." + look: "The re-parent's rows affected: one per part in the moved subtree." + converged: + why: "Structure lives in edge rows: CONNECT BY explodes an assembly, and a property graph over the same rows answers where-used. The recursive walk costs about 30% more per explosion than a prefix scan in the deck's model; a re-parent is one edge row." + look: "The re-parent reports 1 row affected, and the re-explosion shows the wheelset under P-1200." + measure: + why: "The ratio is the redo for rewriting every descendant's path divided by the redo for one edge update. It grows in step with the subtree, about 1.3 KB per moved part, while the edge update stays near 1 KB." + look: "Your dot at 3 moved parts, near 4×, on a reference line that reaches about 4,000× at 3,000 parts." +measure: + x_label: "parts in the moved subtree" + lab_x: 3 + deck_slides: "30–33" + calibration: + - { x: 3, ratio: 4.23 } # doc 3972 B, conv 940 B + - { x: 30, ratio: 40.92 } # doc 39448 B, conv 964 B + - { x: 300, ratio: 409.29 } # doc 394556 B, conv 964 B + - { x: 3000, ratio: 3997.97 } # doc 3949992 B, conv 988 B +--- +# Pattern 05: Tree / Hierarchy + +**Manufacturing.** Explode a bill of materials (read a whole subassembly's +components fast) and ask "what is this part used in?" + +The deck's scenario: an equipment maker with 1.2 million parts, up to 12 levels +deep. About 2 million explosions a day (planning, costing, the service portal), +about 5,000 where-used questions a day plus recall bursts, and about 200 +engineering change orders (ECOs) a day, many of which re-parent a subassembly of +10 to 40,000 parts. The numbers are illustrative; the shape is universal. + +> Store the edge, derive the position. + +## The document bet + +The materialized-path pattern stores each component with its chain of ancestors as a +string: `/P-1000/P-1100/P-1110`. A subtree read becomes a **left-anchored prefix +scan** (`LIKE '/P-1000/P-1100%'`): an index range scan, O(log n). That is the +cheapest explosion there is, and it is the right instinct for a read-heavy day. + +## Where the needle flips + +Two edges, one on write and one on read: + +- **Write.** The path *encodes position*, so moving one subassembly rewrites the path + of **every descendant.** Re-parent a wheelset and you rewrite the wheelset, the + wheel, the spoke, every node beneath it: write amplification **proportional to the + size of the subtree** for a single logical move. +- **Read.** A genuine graph question ("which assemblies use this wheel?"): upward, + multi-parent, N hops, is something a *downward prefix string simply cannot express.* + Share one component across two assemblies and the materialized path breaks: a node + now has more than one path. Teams usually bolt on a `usedIn` parent list, a second + structure the application has to keep in sync by hand. + +The path stores a derived fact (position) in every node, and derived facts you +store are facts you have to rewrite. + +Put numbers on it (the deck's illustrative cost model, not a benchmark): charge +each explosion 1 unit via the prefix scan and about 30% more via a recursive walk, +and charge each document or row rewrite 3 units. **Break-even sits near 1,000 parts +per move.** Below that, or for a tree that rarely moves, the path's day is cheaper. +A 10,000-part module move makes the path's day about 3× the edge model's; at +40,000 parts it is about 10×. Where-used is left out of the model, which favours +the path; counting it only moves the break-even left. + +## The converged softening + +Split the two questions the path was trying to answer at once. In the canonical +form (the logical model) each parent → child link is one relationship; the +relational projection stores it as one edge row, and position is derived by +walking the edges. + +- **Structure lives in adjacency edges** (`parent_id → child_id`). Re-parenting is a + **single edge update**: the subtree follows automatically, because position is not + baked into every node. Reorg write cost drops from O(subtree) to **O(1)**. +- **Subtree / explosion reads use native recursion** through the SQL access surface: + `CONNECT BY` streams the frontier in one pass. (Emit `CONNECT BY` or `GRAPH_TABLE` + for hierarchy, never a recursive CTE, which dams each level into an intermediate + relation the optimizer can barely optimize.) The walk is not free: it goes level + by level and costs more per explosion than a prefix scan, about 30% in the deck's + model. +- **Genuine graph questions use SQL/PGQ `GRAPH_TABLE`** over the *same rows*: the + where-used, multi-parent, N-hop DAG the prefix string could not express. No separate + graph database to sync, no `usedIn` array to maintain. + +| | Materialized path | Edges + graph | +|---|---|---| +| ECO re-parent | 1 document per part in the subtree | 1 edge row | +| Explode an assembly | 1 prefix range scan | `CONNECT BY` pass (about +30%) | +| Where-used | second structure, app-synced | `GRAPH_TABLE`, same rows | + +The write the path made expensive (reorg) becomes one row, and the read the path +could not do at all (where-used across shared components) becomes one graph match, +paid for with a real recursive-read premium on every explosion. + +## When the path is still the right choice + +A tree that rarely moves and is read one way (a product catalogue, an org chart +snapshot), or one whose moves stay well under about 1,000 parts: keep the +materialized path. Its prefix scan is the cheapest explosion there is, and with +few or small moves there is little write amplification to pay for. Store the edge +when engineering reshapes large subtrees every day or the reverse question matters. + +## In this folder + +| File | What it shows | +|---|---| +| `01-document-model.sql` | Materialized paths; a re-parent rewrites every descendant's path | +| `02-converged.sql` | Adjacency edges (reorg = one row) + `CONNECT BY` explosion + `GRAPH_TABLE` where-used | + +## Validated + +Both scripts run clean on **Oracle AI Database 26ai Free (`23.26.3-faststart`)**. This +pattern stays pure-SQL: adjacency/graph traversal (`CONNECT BY` / `GRAPH_TABLE`) has no +single-collection document shape, so there is no MongoDB API lane here. +Run them with `../../run.sh 05-tree-hierarchy`. diff --git a/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/calibrate.sql b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/calibrate.sql new file mode 100644 index 0000000..e8de268 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/calibrate.sql @@ -0,0 +1,19 @@ +-- Calibration resize for Measure it: the wheelset subtree (P-1100, P-1110, P-1111) holds :n +-- parts on BOTH sides -- the lab's 3 plus :n - 3 generated spokes under the wheel P-1110. +DELETE FROM tr_bom_doc WHERE JSON_VALUE(data, '$._id') LIKE 'P-S%'; +INSERT INTO tr_bom_doc (data) +SELECT JSON_OBJECT('_id' VALUE 'P-S' || LPAD(k, 6, '0'), 'name' VALUE 'Spoke ' || k, + 'path' VALUE '/P-1000/P-1100/P-1110/P-S' || LPAD(k, 6, '0') RETURNING JSON) +FROM (SELECT LEVEL AS k FROM dual CONNECT BY LEVEL <= :n) +WHERE k <= :n - 3; +DELETE FROM tr_bom_edges WHERE child_id LIKE 'P-S%'; +DELETE FROM tr_parts WHERE part_id LIKE 'P-S%'; +INSERT INTO tr_parts (part_id, name) +SELECT 'P-S' || LPAD(k, 6, '0'), 'Spoke ' || k +FROM (SELECT LEVEL AS k FROM dual CONNECT BY LEVEL <= :n) +WHERE k <= :n - 3; +INSERT INTO tr_bom_edges (parent_id, child_id, qty) +SELECT 'P-1110', 'P-S' || LPAD(k, 6, '0'), 1 +FROM (SELECT LEVEL AS k FROM dual CONNECT BY LEVEL <= :n) +WHERE k <= :n - 3; +COMMIT; diff --git a/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/doc-shape.svg b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/doc-shape.svg new file mode 100644 index 0000000..4306c86 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/doc-shape.svg @@ -0,0 +1 @@ +parts · one document per part{"_id": "P-88213","name": "pump seal kit","rev": "C","unitCost": 14.20,"path": "root.A-1.SA-7.HP-3","depth": 4,"usedIn": ["HP-3", "HP-9", "VB-12"],"rollupCost": 14.20,"supplier": "S-417"}path = positionan ECO above it rewritesthis and every descendantsecond structure forwhere-used: parent list,kept in sync by app code \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/erd.svg b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/erd.svg new file mode 100644 index 0000000..031e914 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/erd.svg @@ -0,0 +1 @@ +CANONICAL FORM: EDGES, NOT BAKED PATHSTHE SAME MOVE, AS PATHSPARTPKpart_idvarchar2namevarchar2revvarchar2unit_costnumberFKsupplier_idvarchar2BOM_EDGEPKedge_idnumberparent_idvarchar2FKchild_idvarchar2qtynumberECOPKeco_idvarchar2FKedge_idnumberfrom_parentvarchar2released_attimestampparent → child · a part can have many parents (DAG)ECO-4471 · re-parent SA-7A-1 → A-21SA-7 itself10 – 40kdescendants≈ subtree-size rewritesper ECO · not one transactionAdjacency edge: 1 row updatethe subtree follows · atomicthe column an ECO writes: structure, not positionimmutable change history: safe to copy \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/flow.svg b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/flow.svg new file mode 100644 index 0000000..b2add58 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/flow.svg @@ -0,0 +1 @@ +RELATIONAL PROJECTIONSAME ROWS, TWO READSCONSUMERSbom_edgesparent → childECO writes hereparts1.2M rowsCONNECT BYexplode · roll upone recursive passbom_graphSQL/PGQ graphwhere-used, N hopsPlanningSQL · explodeCost rollupSQL · nightlyRecall deskwhere-usedSQL/PGQ \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/model-curve.svg b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/model-curve.svg new file mode 100644 index 0000000..78c787c --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/05-tree-hierarchy/figures/model-curve.svg @@ -0,0 +1 @@ +KNOB SETTINGS · THIS WORKLOAD1 · DiversityHigh: 4 questionsexplode, roll up, where-used,recall impact2 · Read / writeRead-heavy~2M explosions/day against~200 ECOs/day3 · Update localityLands on every descendantone re-parent rewrites10 to 40,000 part pathsDAILY WORK vs PARTS PER MOVED SUBTREE · illustrative modelpath is cheaperedges are cheaper2M5M10M20M101001k10kparts in the moved subtree (log)work units per day (log)break-even ≈ 1,000 partsroutine ECO · 50module move · 10kMaterialized pathAdjacency edges · CONNECT BY, +30% read \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/06-outlier/01-document-model.sql b/labs/converged-modeling-patterns/patterns/06-outlier/01-document-model.sql new file mode 100644 index 0000000..a637876 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/06-outlier/01-document-model.sql @@ -0,0 +1,114 @@ +-- ============================================================================ +-- Pattern 06 · Outlier (whale documents) · THE DOCUMENT MODEL (the starting point) +-- Industry: financial services (brokerage). Access pattern: open an advisor and +-- read their book of clients, top 100 by AUM, as one document. +-- +-- The document move: embed the client book on the advisor doc. This is right for +-- the TYPICAL advisor: the median book is 180 clients, 99% are under 500, and +-- one document read is about 3x cheaper than a range scan at the median (the +-- deck's illustrative model; break-even is near 3,200 clients). +-- +-- THE OUTLIER PROBLEM: about 120 institutional and team advisors carry books of +-- 20,000 to 150,000 clients. Their document blows past the 16 MB ceiling, so the +-- pattern adds a `hasExtras` flag, spills the tail into OVERFLOW documents, and +-- every reader must now branch: "if hasExtras, go fetch the overflow and stitch +-- it back." That is the 16 MB document limit LEAKING INTO YOUR APPLICATION CODE. The +-- pattern does not solve the outlier -- it ADMITS the model breaks for the fat +-- tail and pushes the special case up into every consumer. +-- +-- AND THE WRITES: until it spills, the whale's book is one growing document, so +-- adding a client is a read-modify-write of the whole 800-client book. Spilling +-- to overflow is itself cheap -- the outlier pattern costs you the growth +-- rewrite of the document before the spill, plus a branch in every reader. +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @01-document-model.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables + WHERE table_name IN ('OL_ADVISOR_DOC','OL_ADVISOR_OVERFLOW') + ORDER BY table_name) LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' PURGE'; + END LOOP; +END; +/ + +CREATE JSON COLLECTION TABLE ol_advisor_doc; +CREATE JSON COLLECTION TABLE ol_advisor_overflow; + +-- Typical advisor: whole book embedded, hasExtras=false. The model works. +INSERT INTO ol_advisor_doc VALUES (JSON('{"_id":"A-001","name":"Grace Advisor","hasExtras":false, + "clients":[{"clientId":"C-001","aum":250000}, + {"clientId":"C-002","aum":180000}, + {"clientId":"C-003","aum":420000}]}')); + +-- Whale advisor: stands in for a 148K-client institutional book. 800 clients are +-- already embedded (the x800 skew over the typical book; ~30 KB of JSON), +-- generated, not pasted. It has not spilled yet: hasExtras=false. +INSERT INTO ol_advisor_doc +SELECT JSON_OBJECT('_id' VALUE 'A-900', 'name' VALUE 'Institutional Desk', 'hasExtras' VALUE FALSE, + 'clients' VALUE clients RETURNING JSON) +FROM (SELECT JSON_ARRAYAGG(JSON_OBJECT('clientId' VALUE 'C-' || TO_CHAR(900000 + LEVEL), + 'aum' VALUE 9000000 - (LEVEL - 1) * 10000) + ORDER BY LEVEL RETURNING JSON) AS clients + FROM dual CONNECT BY LEVEL <= 800); +COMMIT; + +-- @step Add one client to the whale's embedded book +-- @note Appending one client is a read-modify-write of the whole 800-client book: the growth rewrite you pay on every add until the document spills. +-- @why Update locality is the hot knob: writes land in the biggest documents on the platform, and appending one client is a read-modify-write of the whole 800-client book. +-- @look Rows affected is 1; Measure it shows the redo of rewriting the whole book next to one client row. +-- @figure doc-shape.svg The advisor document as built: the client book embedded as an array, with a hasExtras flag for books that overflow +-- @measure add-client +UPDATE ol_advisor_doc +SET data = JSON_TRANSFORM(data, APPEND '$.clients' = JSON_OBJECT('clientId' VALUE 'C-900801', 'aum' VALUE 1000000)) +WHERE JSON_VALUE(data,'$._id') = 'A-900'; +COMMIT; + +-- Once the book outgrows the document (the real whale's 148K clients cannot fit +-- under 16 MB; this 800-client book stands in for it), the tail spills into an +-- overflow document. +-- @step Spill the next clients to an overflow document +-- @note Spilling to overflow is itself cheap: the outlier pattern costs you the growth rewrite of the document before the spill, plus a branch in every reader. +-- @why Update locality: the spill write itself is small. What the pattern costs is the growth rewrite before it and the branch every reader gains after it. +-- @look Rows affected is 1, a new overflow document holding 2 clients. +INSERT INTO ol_advisor_overflow VALUES (JSON('{"_id":"A-900#part-2","advisorId":"A-900", + "clients":[{"clientId":"C-900802","aum":990000}, + {"clientId":"C-900803","aum":980000}]}')); +COMMIT; + +-- @step Flag the whale: hasExtras = true +-- @note From here on, every reader has to check this flag and go fetch the overflow. +-- @why Diversity is low, yet every consumer of the book still has to learn this flag: the portal, the batch and the AI assistant's context builder. +-- @look Rows affected is 1, and has_extras on the whale read now shows true. +UPDATE ol_advisor_doc +SET data = JSON_TRANSFORM(data, SET '$.hasExtras' = TRUE) +WHERE JSON_VALUE(data,'$._id') = 'A-900'; +COMMIT; + +-- The read for a TYPICAL advisor: one document, done. +-- @step Read a typical advisor (one document) +-- @note The ordinary case: one document, done. No branch needed. +-- @why Read/write is the pattern's case: for the 180-client median, one document read is about 3× cheaper than a query in the deck's model. +-- @look has_extras is false and clients_inline holds A-001's whole book. +SELECT JSON_VALUE(data,'$._id') AS advisor, + JSON_VALUE(data,'$.hasExtras') AS has_extras, + JSON_QUERY(data,'$.clients' RETURNING CLOB) AS clients_inline +FROM ol_advisor_doc WHERE JSON_VALUE(data,'$._id') = 'A-001'; + +-- The read for the WHALE: the application must notice hasExtras and go stitch the +-- overflow back in. The 16 MB limit is now branching logic in every consumer +-- (including the LLM's context builder). +-- @step Read the whale advisor (branch on hasExtras) +-- @note Every reader must notice hasExtras and go stitch the overflow back in: the 16 MB limit is now application logic. Shows counts and the first 5 embedded clients, not the whole 800-client array. +-- @why Update locality's skew shows up in the reader: the 16 MB cap has become a branch, and the rare path is the one least tested. +-- @look clients_embedded is 801 and clients_in_overflow is 2 if you ran the two cards above (800 and 0 on fresh data): one advisor, read from two collections and stitched together. +SELECT JSON_VALUE(d.data,'$._id') AS advisor, + JSON_VALUE(d.data,'$.hasExtras') AS has_extras, + JSON_VALUE(d.data,'$.clients.size()' RETURNING NUMBER) AS clients_embedded, + JSON_QUERY(d.data,'$.clients[0 to 4]' RETURNING CLOB WITH ARRAY WRAPPER) AS clients_embedded_first5, + JSON_VALUE(o.data,'$.clients.size()' RETURNING NUMBER) AS clients_in_overflow, + JSON_QUERY(o.data,'$.clients' RETURNING CLOB) AS clients_overflow +FROM ol_advisor_doc d +LEFT JOIN ol_advisor_overflow o + ON JSON_VALUE(o.data,'$.advisorId') = JSON_VALUE(d.data,'$._id') +WHERE JSON_VALUE(d.data,'$._id') = 'A-900'; diff --git a/labs/converged-modeling-patterns/patterns/06-outlier/02-converged.sql b/labs/converged-modeling-patterns/patterns/06-outlier/02-converged.sql new file mode 100644 index 0000000..1716bdb --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/06-outlier/02-converged.sql @@ -0,0 +1,119 @@ +-- ============================================================================ +-- Pattern 06 · Outlier (whale documents) · THE CONVERGED ALTERNATIVE +-- "The outlier is just more rows, and the optimizer plans for it." +-- +-- In the canonical form a client is its own entity, related to one advisor. +-- Here it is stored in the relational projection: rows in one table, keyed by +-- advisor_id. There is no special document shape, no hasExtras flag, no overflow +-- collection, no branch in the reader. Where a consumer wants the book as a +-- document, a duality view produces it from the rows; the institutional whale +-- is the SAME query -- it just returns more rows, and you page them with +-- FETCH FIRST / OFFSET. One table, one index, one query for every advisor. +-- +-- The 16 MB document ceiling never enters the picture, because the book was +-- never one physical document. The fat tail stops being an application special +-- case and becomes ordinary pagination over ordinary rows. +-- +-- THE TRADEOFF: reads are not free. Every book read is an index descent plus +-- assembled rows, about 3x the work of one stored document at the 180-client +-- median in the deck's model. That read cost buys one code path for all +-- advisors and writes that do not grow with the book. If a list has a hard +-- upper bound (five beneficiaries, ten watchlists), embedding is still right. +-- Run: sqlplus cmp_user/CmpUser2026@localhost:1521/FREEPDB1 @02-converged.sql +-- ============================================================================ + +BEGIN + FOR r IN (SELECT table_name FROM user_tables + WHERE table_name IN ('OL_CLIENTS','OL_ADVISORS') + ORDER BY table_name) LOOP + EXECUTE IMMEDIATE 'DROP TABLE ' || r.table_name || ' CASCADE CONSTRAINTS PURGE'; + END LOOP; +END; +/ + +CREATE TABLE ol_advisors ( + advisor_id VARCHAR2(12) PRIMARY KEY, + name VARCHAR2(120) NOT NULL +); + +CREATE TABLE ol_clients ( + client_id VARCHAR2(12) PRIMARY KEY, + advisor_id VARCHAR2(12) NOT NULL REFERENCES ol_advisors(advisor_id), + aum NUMBER(18,2) NOT NULL +); +-- The access path that makes a whale's book a range scan, not a scan-the-world. +CREATE INDEX ol_ix_client_advisor ON ol_clients (advisor_id, aum DESC); + +INSERT INTO ol_advisors VALUES ('A-001','Grace Advisor'); +INSERT INTO ol_advisors VALUES ('A-900','Institutional Desk'); +INSERT INTO ol_clients VALUES ('C-001','A-001',250000); +INSERT INTO ol_clients VALUES ('C-002','A-001',180000); +INSERT INTO ol_clients VALUES ('C-003','A-001',420000); +-- The "whale" -- stands in for a 148K-client institutional book: the same 800 +-- clients the document model embeds, generated, not pasted. Just more rows. +INSERT INTO ol_clients (client_id, advisor_id, aum) +SELECT 'C-' || TO_CHAR(900000 + LEVEL), 'A-900', 9000000 - (LEVEL - 1) * 10000 +FROM dual CONNECT BY LEVEL <= 800; +COMMIT; +-- @step Add one more client to the whale's book +-- @note Just another row: no ceiling, no overflow shard, no reader branch, and the same cost at 800 clients as at 3. +-- @why Update locality stops scaling with the book: a new client is one row and one index entry, the same cost at 3 clients or 148,000. +-- @look Rows affected is 1; Measure it shows redo near 1.2 KB at any book size. +-- @figure erd.svg Desk, advisor and client in canonical form, beside the typical one-document book and the institutional book split into overflow documents +-- @measure add-client +INSERT INTO ol_clients VALUES ('C-900801','A-900',1000000); +COMMIT; + +-- Where the document model spills to overflow, the converged book just grows. +-- @step Add the next two clients to the whale's book +-- @note No spill: just two more rows. +-- @why Update locality: where the document model spilled to a second collection, the book just gets two more rows. +-- @look Rows affected is 2, with no flag and no overflow write. +INSERT INTO ol_clients (client_id, advisor_id, aum) +SELECT 'C-900802', 'A-900', 990000 FROM dual UNION ALL +SELECT 'C-900803', 'A-900', 980000 FROM dual; +COMMIT; + +-- The typical advisor as a document -- assembled by the duality view over the +-- SAME rows on every read. No special case, no flag. (For a bounded book this is +-- the whole doc.) +CREATE OR REPLACE JSON RELATIONAL DUALITY VIEW ol_advisor_dv AS +SELECT JSON { + '_id' : a.advisor_id, + 'name' : a.name WITH UPDATE, + 'clients' : [ SELECT JSON { 'clientId' : c.client_id, + 'aum' : c.aum WITH UPDATE } + FROM ol_clients c WITH INSERT UPDATE DELETE + WHERE c.advisor_id = a.advisor_id ] +} FROM ol_advisors a WITH INSERT UPDATE DELETE; + +-- @step Read the typical advisor as a projected document +-- @note Same shape the document model served: no flag, no branch. +-- @why Diversity: the duality view projects the same advisor document from rows, assembling the client array on read rather than storing it. +-- @look typical_advisor_document shows A-001's clients array, with no hasExtras field. +SELECT JSON_SERIALIZE(data PRETTY) AS typical_advisor_document +FROM ol_advisor_dv WHERE JSON_VALUE(data,'$._id') = 'A-001'; + +-- The WHALE -- exact same table, exact same optimizer. You do not stitch an +-- overflow collection; you page ordinary rows. Top 3 holdings for the desk: +-- @step Read the whale's top 3 holdings +-- @note Exact same table, exact same optimizer: no overflow collection to stitch back in. +-- @why Read/write, priced honestly: a top-N range scan on (advisor_id, aum DESC) reads index entries and assembles rows on every read, but the same amount at 180 clients or 148,000. +-- @look The 3 largest holdings by aum, from ol_clients with no overflow to stitch. +SELECT client_id, aum +FROM ol_clients +WHERE advisor_id = 'A-900' +ORDER BY aum DESC +FETCH FIRST 3 ROWS ONLY; + +-- The next page -- OFFSET, same index range scan. This is the whole "outlier" +-- story on a converged engine: pagination, not a special document shape. +-- @step Page to the next 3 holdings +-- @note OFFSET, same index range scan: ordinary pagination, not a special document shape. +-- @why Update locality's skew becomes a statistic, not a code path: the next page is the same query with OFFSET. +-- @look Holdings 4 to 6 by aum, from the same index range. +SELECT client_id, aum +FROM ol_clients +WHERE advisor_id = 'A-900' +ORDER BY aum DESC +OFFSET 3 ROWS FETCH NEXT 3 ROWS ONLY; diff --git a/labs/converged-modeling-patterns/patterns/06-outlier/03-parity.js b/labs/converged-modeling-patterns/patterns/06-outlier/03-parity.js new file mode 100644 index 0000000..3099911 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/06-outlier/03-parity.js @@ -0,0 +1,42 @@ +// ============================================================================ +// Pattern 06 · Outlier · CROSS-API PARITY (mongosh) +// The SAME typical-advisor document (its client array assembled from ol_clients +// rows) read through the SQL duality view (ol_advisor_dv, injected as $SQL_RESULT) +// and through the MongoDB API. Byte-equal after canonicalizing JSON and ignoring +// _metadata. Non-zero exit on mismatch. +// ============================================================================ +function canon(x) { + if (x === null || x === undefined) return null; + if (Array.isArray(x)) return x.map(canon).sort((a, b) => + JSON.stringify(a) < JSON.stringify(b) ? -1 : 1); + if (typeof x === "object") { + const cn = x.constructor && x.constructor.name; + if (x._bsontype || /Decimal128|Double|Long|Int32/.test(cn || "")) return Number(x.toString()); + const o = {}; + for (const k of Object.keys(x).sort()) { if (k === "_metadata") continue; o[k] = canon(x[k]); } + return o; + } + return x; +} +const eq = (a, b) => JSON.stringify(canon(a)) === JSON.stringify(canon(b)); + +const viaSql = JSON.parse(process.env.SQL_RESULT || "null"); +if (!viaSql) { print("[FAIL] SQL_RESULT not provided by run.sh"); quit(1); } + +// @step Read the projected document over the MongoDB API +// @note Same typical-advisor document the SQL lane read: the client array projects from ol_clients, whichever wire protocol you read it over. +// @why Diversity: the CRM keeps find() over the MongoDB API while the book screen runs its top-N query in SQL over the same client rows, so there is one book and no branch. +// @look The A-001 document's clients array matches the SQL projection, with no hasExtras flag. +// @figure flow.svg Advisor and client tables feed a duality view for document consumers and one top-N query for the book screen +const viaMongo = db.ol_advisor_dv.findOne({ _id: "A-001" }); +if (!viaMongo) { print("[FAIL] MongoDB API returned no document for A-001"); quit(1); } + +if (!eq(viaSql, viaMongo)) { + print("[FAIL] cross-API document mismatch for advisor A-001"); + print(" SQL lane: " + JSON.stringify(canon(viaSql))); + print(" Mongo lane: " + JSON.stringify(canon(viaMongo))); + quit(1); +} +print("[PASS] ol_advisor_dv document byte-equal across SQL and MongoDB API (A-001)"); +print(" clients projected via Mongo lane = " + viaMongo.clients.length); +quit(0); diff --git a/labs/converged-modeling-patterns/patterns/06-outlier/README.md b/labs/converged-modeling-patterns/patterns/06-outlier/README.md new file mode 100644 index 0000000..28d8b9b --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/06-outlier/README.md @@ -0,0 +1,133 @@ +--- +title: Outlier +industry: Financial services (brokerage) +deck: "34–37" +problem: >- + Advisor documents embed their client book, which is right for the 180-client median. About + 120 institutional and team books of 20,000 to 150,000 clients break the 16 MB cap, so the + Outlier pattern adds overflow documents and a second code path in the application. +knobs: + - { name: Diversity, setting: "Low: one book screen", help: "The portal and the batch read the book the same way, top clients by AUM, so one shape serves both." } + - { name: Read / write, setting: "Read-heavy by day", help: "About 2,000 book reads a second at the open, and one AUM refresh storm a night." } + - { name: Update locality, setting: "Skew ×800, p50 → max", hot: true, help: "The largest book is about 800× the median and takes the nightly refresh and every transition: embedded is about 3× cheaper at the 180-client median, about 50× dearer at a 150,000-client book, with break-even near 3,200." } +help: + tabs: + document: + why: "The starting point: embed the client book on the advisor document, which is right for the 99% of advisors under 500 clients. Books that outgrow the 16 MB cap spill into overflow documents behind a hasExtras flag, and every reader gains a branch." + look: "The whale read's has_extras, clients_embedded and clients_in_overflow: one advisor, two places to look." + converged: + why: "Clients are rows with an (advisor_id, aum DESC) index, so every book is the same top-N query whether it holds 180 clients or 148,000. At the median the single document was cheaper, about 3× in the deck's model; the rows buy one code path." + look: "The insert reports 1 row affected, and the whale's top 3 and next 3 come from the same query with OFFSET." + mongo: + why: "The CRM reads the book as a document over the MongoDB API and the AI assistant reads the same projection over SQL/JSON, both from the client rows the book screen queries." + look: "The returned A-001 document carries its clients array and no hasExtras flag." + measure: + why: "The ratio is the redo for appending one client to the embedded book divided by the redo for one client row. The document side rewrites the whole book, so the ratio grows with the clients already embedded; the row insert stays near 1.2 KB." + look: "Your dot at 800 clients, near 17× (about 20 KB of redo against 1.2 KB)." +measure: + x_label: "clients embedded in the advisor's book" + lab_x: 800 + deck_slides: "34–37" + calibration: + - { x: 10, ratio: 1.65 } # doc 2016 B, conv 1224 B + - { x: 100, ratio: 5.01 } # doc 6136 B, conv 1224 B + - { x: 800, ratio: 16.96 } # doc 20760 B, conv 1224 B + - { x: 2000, ratio: 43.39 } # doc 48768 B, conv 1124 B +--- +# Pattern 06: Outlier (whale documents) + +**Financial services (brokerage).** Open an advisor and read their book of clients, +top 100 by AUM, as one document. The platform has about 40,000 advisors and 17 million +clients (illustrative numbers; the shape of the distribution is the point). + +> The outlier is just more rows, and the optimizer plans for it. + +## The document bet + +The Outlier pattern tunes the document for the *typical* record and special-cases the +fat tail. Embed the client book on the advisor document. The median advisor has 180 +clients and 99% have fewer than 500, so for almost everyone the book is one document of +roughly 72 KB: one read, sorted in the app, done. For those advisors it genuinely beats +anything else, and that is worth saying plainly. + +## Where the needle flips + +About 120 institutional and team advisors carry **20,000 to 150,000-client books.** +At roughly 400 bytes a client the **16 MB ceiling** arrives near 40,000 clients, so the +team spills at 20,000 per document: the pattern adds a `hasExtras` flag, parks the tail +in **overflow documents** (seven of them for a 148,000-client book), and now **every +reader must branch**: *"if hasExtras, go fetch the overflow and stitch it back."* That +is the 16 MB document limit **leaking into your application code**: the portal, the +nightly batch, and the AI assistant's context builder all have to know about the +special case. + +The knobs show why this one flips on the distribution, not the average. Diversity is +low (one screen reads the book one way) and the workload is read-heavy by day, about +2,000 book reads a second at the open; on those two knobs alone you would embed and move +on. Update locality is the hot knob: the largest book is about 800 times the median, and +the nightly AUM refresh and every advisor transition land inside those largest +documents. You design for the p50 and pay at the p99.9. + +In the deck's illustrative cost model (not a benchmark), for a top-100 read the embedded +book is **about 3 times cheaper** at the 180-client median, break-even is **near 3,200 +clients**, and a 148,000-client team book is **about 50 times more work**: eight +fetches, every client decoded, and a merge-sort in the application, on a code path that +only the 120 largest books ever exercise. + +The pattern does not solve the outlier. It **admits the model breaks** for the fat +tail and pushes the special case up into every consumer. + +The writes pay too. Until it spills, the whale's book is one growing document: in the +lab, advisor A-900 already embeds 800 clients, and adding one more is a +read-modify-write of the whole book: about 20 KB of redo against roughly 1.2 KB for one +client row, near 17 times in this lab (the deck's 26ai Free run measured about 20 +times). Spilling to overflow is itself cheap: the outlier pattern costs you the growth +rewrite of the document before the spill, plus a branch in every reader. + +## The converged softening + +Start from the canonical form, the logical model: desk, advisor and client, where a +client is its own entity related to one advisor. The outlier exists only because the +document projection draws a physical boundary around the book. Store the client in a +different projection, **rows in one table** keyed by `advisor_id`, and there is no +special shape: + +- Where a consumer wants the book as a document (the CRM over the MongoDB API, the AI + assistant over SQL/JSON), a **duality view** produces it from the rows: no flag, no + branch. +- The book screen runs one top-N query over the `(advisor_id, aum DESC)` index and + **pages** with `FETCH FIRST` / `OFFSET`. The institutional **whale** is the *same + table, same query*: the optimizer plans a stopped range scan that reads about 100 + index entries whether the book holds 180 clients or 148,000. Cardinality is a + statistic, not a code path. +- Moving a book to another advisor is one `UPDATE` of `advisor_id` in one transaction, + with no receiving document that might itself cross the cap. + +The 16 MB ceiling never enters the picture, because the book was never one physical +document. + +**The tradeoff, stated plainly.** The rows are not free to read. Every book read is a +query, an index descent and 100 assembled rows, and at the median that is about 3 times +the work of one stored document in the deck's model. What that read cost buys is one +code path for all 40,000 advisors, and writes that no longer scale with the size of the +book. + +**When the document is still the right choice.** If the list has a hard upper bound +(ten saved watchlists, five beneficiaries), embed it. The Outlier pattern, and the +argument against it, only matters when the bound is a guess. + +## In this folder + +| File | What it shows | +|---|---| +| `01-document-model.sql` | Embedded book (800-client whale) + growth append + `hasExtras` flag + overflow collection + reader branch | +| `02-converged.sql` | One clients table (the same 800-client whale as rows); a duality view produces the typical book as a document; the whale is paged rows | +| `03-parity.js` | Reads `ol_advisor_dv` via the MongoDB API and asserts byte-equality with the SQL lane | +| `_capture.sql` | Helper: emits the SQL-lane document for the parity check | + +## Validated + +Both lanes run clean on **Oracle AI Database 26ai Free (`23.26.3-faststart`)**. Cross-API +parity passes: the typical-advisor document (client array projected from rows) is +identical through SQL and the MongoDB API. +Run with `../../run.sh 06-outlier`. diff --git a/labs/converged-modeling-patterns/patterns/06-outlier/_capture.sql b/labs/converged-modeling-patterns/patterns/06-outlier/_capture.sql new file mode 100644 index 0000000..83f8ac8 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/06-outlier/_capture.sql @@ -0,0 +1,7 @@ +-- Capture the typical advisor document as seen from SQL (duality view +-- ol_advisor_dv), serialized to one JSON string, for cross-API parity. +SET HEADING OFF PAGESIZE 0 FEEDBACK OFF VERIFY OFF TRIMSPOOL ON +SET LINESIZE 32767 LONG 100000000 LONGCHUNKSIZE 1000000 +SELECT JSON_SERIALIZE(data RETURNING CLOB) +FROM ol_advisor_dv +WHERE JSON_VALUE(data, '$._id') = 'A-001'; diff --git a/labs/converged-modeling-patterns/patterns/06-outlier/calibrate.sql b/labs/converged-modeling-patterns/patterns/06-outlier/calibrate.sql new file mode 100644 index 0000000..e54c84c --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/06-outlier/calibrate.sql @@ -0,0 +1,16 @@ +-- Calibration resize for Measure it: the whale advisor A-900 has :n clients on BOTH sides, +-- generated as the lab's 800 are. Ids start at C-700001 so they never collide with the +-- measured C-900801 when :n exceeds 800. +DELETE FROM ol_advisor_doc WHERE JSON_VALUE(data, '$._id') = 'A-900'; +INSERT INTO ol_advisor_doc +SELECT JSON_OBJECT('_id' VALUE 'A-900', 'name' VALUE 'Institutional Desk', 'hasExtras' VALUE FALSE, + 'clients' VALUE clients RETURNING JSON) +FROM (SELECT JSON_ARRAYAGG(JSON_OBJECT('clientId' VALUE 'C-' || TO_CHAR(700000 + LEVEL), + 'aum' VALUE 9000000 - (LEVEL - 1) * 1000) + ORDER BY LEVEL RETURNING JSON) AS clients + FROM dual CONNECT BY LEVEL <= :n); +DELETE FROM ol_clients WHERE advisor_id = 'A-900'; +INSERT INTO ol_clients (client_id, advisor_id, aum) +SELECT 'C-' || TO_CHAR(700000 + LEVEL), 'A-900', 9000000 - (LEVEL - 1) * 1000 +FROM dual CONNECT BY LEVEL <= :n; +COMMIT; diff --git a/labs/converged-modeling-patterns/patterns/06-outlier/figures/doc-shape.svg b/labs/converged-modeling-patterns/patterns/06-outlier/figures/doc-shape.svg new file mode 100644 index 0000000..208c4d5 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/06-outlier/figures/doc-shape.svg @@ -0,0 +1 @@ +advisors · one document per book{"_id": "A-900","name": "Institutional Desk 4","bookType": "team","hasExtras": true,"clientCount": 148212,"clients": [{ "id": "C-900001","name": "Harbor Pension Trust","aum": 912400000 },{ "id": "C-900002","aum": 887150000 },... 19,998 more ...]}the 16 MB cap, leaked:128k more clients sitin 7 overflow docsnightly AUM refreshrewrites the whole~8 MB document \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/06-outlier/figures/erd.svg b/labs/converged-modeling-patterns/patterns/06-outlier/figures/erd.svg new file mode 100644 index 0000000..6ac9966 --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/06-outlier/figures/erd.svg @@ -0,0 +1 @@ +CANONICAL FORM: A CLIENT IS ITS OWN ENTITYTHE SAME BOOK, AS DOCUMENTSDESKPKdesk_idvarchar2desk_namevarchar2book_typevarchar2ADVISORPKadvisor_idvarchar2FKdesk_idvarchar2full_namevarchar2regionvarchar2CLIENTPKclient_idvarchar2FKadvisor_idvarchar2full_namevarchar2aumnumberaum_as_ofdateopened_ondateINDEX (advisor_id, aum DESC)any book, any size: one range scanp50 advisor · 180 clients1 document · ~72 KB · one readteam book · 148k clientsadvisor doc: first 20k + hasExtras#part-220k clients#part-320k clients… #part-820k clientsapplication branchif hasExtras: 1 + 7 fetches, merge-sortRelational projection1 query, any booksame index range scan · no branchmutable: rewritten nightly, inside the biggest documentsimmutable: safe to copy anywhere \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/06-outlier/figures/flow.svg b/labs/converged-modeling-patterns/patterns/06-outlier/figures/flow.svg new file mode 100644 index 0000000..f07e14b --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/06-outlier/figures/flow.svg @@ -0,0 +1 @@ +RELATIONAL PROJECTIONSAME ROWS, TWO READSCONSUMERSadvisorsone row eachclientsidx advisor_id,aum DESCadvisor_dvduality viewbook as a documenttop-N queryFETCH FIRST ·OFFSET pagesCRMMongo API find()AI assistantSQL/JSONBook screenSQL · any size \ No newline at end of file diff --git a/labs/converged-modeling-patterns/patterns/06-outlier/figures/model-curve.svg b/labs/converged-modeling-patterns/patterns/06-outlier/figures/model-curve.svg new file mode 100644 index 0000000..4cecc6f --- /dev/null +++ b/labs/converged-modeling-patterns/patterns/06-outlier/figures/model-curve.svg @@ -0,0 +1 @@ +KNOB SETTINGS · THIS WORKLOAD1 · DiversityLow: one book screenportal and batch read thebook the same way2 · Read / writeRead-heavy by day~2k reads/s at the open;one refresh storm a night3 · Update localitySkew ×800, p50 → maxwrites land in the biggestdocuments on the platformWORK PER TOP-100 BOOK READ vs BOOK SIZE · illustrative modelone document is cheaperrange scan is cheaper10301003001k3k101001k10k100kclients per advisor (log)work units per read (log)break-even ≈ 3,200 clientsp50 · 180 clientsteam book · 148kEmbedded book + overflowRows · index range scan, top 100 \ No newline at end of file diff --git a/labs/converged-modeling-patterns/presentations/converged-data-modeling-workshop1.html b/labs/converged-modeling-patterns/presentations/converged-data-modeling-workshop1.html new file mode 100644 index 0000000..db99fd7 --- /dev/null +++ b/labs/converged-modeling-patterns/presentations/converged-data-modeling-workshop1.html @@ -0,0 +1,1523 @@ + + + + + +Model the Domain, Not the Engine + + + + +
+ + +
+
Workshop 1 · Converged Data Modeling · Oracle AI Database 26ai
+

Model the domain, not the engine

+

Every document pattern you know is a workaround for something the engine couldn't do. This session walks the patterns, finds the physics underneath each one, and shows where a converged engine dissolves the tradeoff instead of paying for it.

+
+
3
performance knobs that decide every model
+
12
document patterns, walked and stress-tested
+
1
engine · one transaction · one optimizer
+
90m
lecture; the lab is where you build it
+
+ +
+ + +
+
Act 0 · Framing0:02 – 0:05
+

The choice was never "relational or document"

+
    +
  • Relational vs. document was never a team to join
    • +
  • Model once, project many: the workload picks the shape
    • +
+
+
THE OLD FRAMEA database per workloadDocumentSearchGraphVectorRelationalintegration taxserialize · hop · authconsistency gapcopies disagree meanwhilereframeTHE FRAME FOR TODAYModel once, project manyCanonical formentities · relationshipseach fact oncePROJECTIONS, READ AS SHAPESDocumentGraphVectorRelationalone transaction · one optimizerTODAY'S MAPa modeling session, not a product pitch1Why nowtimeline · UMT2The physicsthree knobs3The pattern walksix industries4How you knowthe bake-offThe rule all day:the workload chooses the shape; measure, don't guess.
+
+ +
+ + +
+
Act 1 · Why converged, why now0:05 – 0:09
+

Data pressure through the ages

+
    +
  • Every era's model was the right bet for its bottleneck
    • +
  • Today's document patterns were bets placed in the NoSQL era
    • +
+
+
Data pressuretimeHumanBrain~100,000 BCLedgers~3000 BCPunchCards1890sMagneticTape1950sBlockStorage1970sRDBMS1980sDistributedNoSQL2000sSQLDocGraphTimeVectorPolyglotPersistence2015+Unified ModelTheory2020sEach peak = data pressure exceeding current technology. Each valley = the solution.
+
+ +
+ + +
+
Act 1 · Why converged, why now0:09 – 0:12
+

Why we ever denormalized: the economics flipped

+
    +
  • Then: disk was expensive, so store each fact once
    • +
  • Now: CPU, network hops and wait time are expensive
    • +
  • Keep one truth; let the engine maintain the read shapes
    • +
+
+
THEN · 1970s–1990sStorage· disk cost real moneyexpensiveCompute & latency· joins were affordablecheap enoughNOW · 2010s →Storage· pennies per gigabytenearly freeCompute & latency· CPU, network hops, user waitexpensiveswapdenormalize → project (engine-maintained)keep one normalized truth; spend cheap storageon read shapes the engine keeps in syncAnswer then: normalizestore each fact once;join at read time to reassemble it
+
+ +
+ + +
+
Act 1 · Why converged, why now0:12 – 0:15
+

Codd already drew the map

+
    +
  • Canonical form: the entities, properties and relationships, each fact once
    • +
  • Tables, documents, graph, vectors: projections of it, kept by the engine
    • +
+
+
+ + + + + + + + + Canonical Form + entities · properties + relationships · the logical model + + + + + + + + + Relationaltables & rows + DocumentJSON · duality views + Graphnodes & edges + Vectorembedding index + + projection = one physical representation of the same facts, maintained by the engine + +
+
+ +
+ + +
+
Act 1 · Why converged, why now0:15 – 0:18
+

Polyglot persistence: five stores, five versions of the truth

+
    +
  • Same fact, five stores, five different sync delays
    • +
  • An AI agent acts on whichever copy it reads
    • +
+
+
One fact changescustomer tier = Gold10:00:00Document storeJSON documentsGoldlag ~msSearch indexfull-text searchGoldlag ~1–15 sVector indexembeddings for RAGSilverstalenext embed jobGraph storerelationshipsGoldlag ~1 sCachehot readsSilverstale10 min expiryintegration taxper copy: serialize · network hop · autheventual consistency gapeach copy catches up on its own clock10:00:05 (five seconds later)AI agent (RAG)retrieval-augmented generationreads the vector copy: SilverreadsActs on “Silver”denies a Gold customer's fee waiverconfident wrong answer(hallucination)
+
+ +
+ + +
+
Act 1 · Why converged, why now0:18 – 0:20
+

Unified Model Theory: the vocabulary for the rest of the day

+
    +
  • UMT = Unified Model Theory: Codd's idea, made operational
    • +
  • Five terms; every decision today has a place in one
    • +
+
+
queries on the access surface render the shapeCanonical Formthe logical modelentitiespropertiesrelationshipsProjectionshow the data is storedtables / rowsdocumentsgraph nodes & edgesAccess Surfacehow consumers askSQL · SQL/JSONSQL/PGQ (graph queries)MongoDB APIProjected Shapewhat the consumer readsa document · a grapha vector resulta rowsetA consumer's queryAccess Dimension: the kind of question it askspointrangeaggregatetraversesimilarityasksModel once.Project many.
+
+ +
+ + +
+
Act 1 · Why converged, why now0:20 – 0:22
+

One query, four models, one transaction

+
    +
  • Relational, JSON, graph and vector in one statement: one transaction, one security boundary
    • +
+
+
+

Find the 5 parties nearest a query embedding, tier from their JSON profile, reachable through ≤3 control hops

+ +
+
+ +
+ + +
+
Act 2 · The physics0:22 – 0:26
+

Three knobs decide every model

+

Strip away the vocabulary and every modeling decision (embed or reference, precompute or query) turns on three measurable quantities. Set these, and the model is half-chosen for you.

+
+
+ + + + + + 1 · Diversity + how many distinct + access patterns? + + + + + + 2 · Read / Write + which side of the + ratio dominates? + + + + + + 3 · Update % + locality + what fraction changes, + and where does it land? + + +
Knob 3 is the one document modeling under-weights: not just how often you write, but where the write lands inside the shape. That's write amplification, and it's the whole ballgame.
+
+
+

Diversity of access · read/write ratio · update % and where it lands. High diversity wants projections. Read-heavy wants precomputation. But knob 3 can veto both, which is where the patterns come from.

+ +
+ + +
+
Act 2 · The physics0:26 – 0:29
+

Why the patterns exist: write amplification

+

Every document pattern in the catalog is, underneath, a reaction to one physical fact: in a document engine, touching one field can rewrite the whole record. Understand this and the patterns stop being folklore.

+
+
+ + + + + + + one large document + + + + update ONE field + + + + + forces + re-serialize the ENTIRE leaf page + + + + + + + + + + + document-level +   lock held + + marches toward +   the 16 MB cap + + +
Copy-on-write leaf pages (WiredTiger): the write cost scales with document size, not with the size of your change. A 2 MB document costs 2 MB to touch, every time, for a one-byte edit.
+
+ +
+

The bigger and hotter the document, the more every write costs. Bucket, Computed, Subset, Pre-Allocation: they all exist to keep documents small and writes cheap. They are write-amplification management, dressed up as design patterns.

+ +
+ + +
+
Act 2 · The physics0:29 – 0:31
+

The breakpoint: when precomputing stops paying

+

Every "should I maintain this precomputed shape?" decision reduces to one inequality. Precompute only when the reads you save outweigh the writes you add.

+
+
+ + + read_freq × read_cost + what precomputing saves + > + write_freq × maint_cost + what precomputing costs + +
When the inequality holds, precompute. When it flips, stop maintaining the shape and query it on demand. The pattern isn't good or bad; it's a bet on this inequality at your cardinality.
+
+
+
+

What moves the right side up: more writes, fatter documents, worse write amplification: a subset trimmed on every write to serve a read that rarely happens.

+
+
+

What a cost-based optimizer does: it lowers read_cost (an index-only Top-N, a hash join, a parallel scan), so the inequality holds at a shape you never had to hand-maintain.

+
+
+
+

A better optimizer moves the breakpoint. Patterns that "must" be precomputed on a document engine can be queried live on a converged one, because the read side just got cheaper.

+ +
+ + +
+
Act 2 · The physics0:31 – 0:33
+

Duality doesn't change the knobs. It changes how they get set.

+

The document catalog offers two resolutions to every modeling tension: embed or reference. Converged adds a third the catalog never had, project it, and it's the one that dissolves the tradeoff instead of trading one cost for another.

+
+
+
+ Resolution 1 +

Embed

+

Fast reads, but you copy the fact. Every copy is write amplification and an update anomaly waiting to happen. Great until knob 3 turns.

+
+
+ Resolution 2 +

Reference

+

No duplication, but the read now needs $lookup or a second round trip, and the document engine has no optimizer to make that join cheap.

+
+
+ Resolution 3 · new +

Project it

+

Model the fact once in canonical form and store it once, as a row; a duality view projects it as a document, live, maintained by the engine. Read shape without write penalty.

+
+
+
+

Same three knobs. But now embed/reference isn't a permanent architectural fork; it's a projection choice you can change without a migration. The needle you couldn't move before is now a setting.

+ +
+ + +
+
Act 3 · The pattern walk0:33 – 0:35
+

Six patterns, six industries, one method

+

Each pattern gets the same four beats. We're not asking "is this pattern good?" We're asking "at what point does its tradeoff flip, and what does converged do at that point?"

+
+
    +
  1. The use case: a real access pattern from a real industry, not a toy.
    2. +
  2. The hard edge: the knob setting where the pattern starts to hurt (almost always knob 3).
    3. +
  3. Where the needle flips: the cardinality or write rate that turns the pattern from clever to costly.
    4. +
  4. The converged softening: the mechanism, with live SQL, that moves the breakpoint.
    5. +
+ + + + + + + + + + +
PatternIndustrySoftened by
Extended ReferenceWealth managementJSON Duality View
ComputedTelecomMV refresh-on-commit · index-only Top-N
Bucket / time-seriesManufacturing IoTPartitioned range · $sql in pipeline
SubsetInsuranceComposite index · FETCH FIRST
Tree / HierarchyManufacturing BOMGRAPH_TABLE · CONNECT BY
OutlierFinancialAdaptive plans · partitioning
+

$sql: Oracle's addition to the MongoDB aggregation pipeline. It runs full SQL as one stage of db.aggregate() (Pattern 3).

+
+

Watch for the refrain: the pattern was a correct bet against a limited engine. Remove the limit and the bet, and its maintenance cost, is optional.

+ +
+ + + +
+
Act 3 · Pattern 1: Extended Reference · A. The problem0:35 – 0:37
+

Wealth management: one advisor card, 2,700 copies

+

Four consumers render every client with their advisor. To make each screen a single read, the team embedded the advisor card in every client and every account document: the Extended Reference pattern. Reads are instant. Now watch where the writes land.

+
+
+
+
2,400 advisors1.9M clients4.6M accounts4 consumers
+
Access patternWhoRateType
Load client 360portal · copilot600/s avg · 12k/s peakread
Render statementsbatch4.6M / nightread
Advisor office / desk changeops~300 / daywrite · fan-out
Regional reorg: reassign advisorsops~8,000 in a day, 2× / yrwrite · fan-out
Update client contactclient~2k / daywrite
+
+
clients · one document per client{"_id": "C-104233","name": "Ada Okafor","segment": "HNW","advisor": {"id": "A-317","name": "Grace Hopper","office": "Leeds","desk": "+44 113 555 0101"},"accounts": [{ "id": "ACC-88", "type": "ISA","advisor": { "id": "A-317","office": "Leeds" } },{ "id": "ACC-91", "type": "GIA", ... }]}advisor card, copied~790 client docscarry this copycopied again peraccount: ~1,920more copies
+
+
+

The reads are the whole reason for the pattern. The rare advisor edit is the whole cost: one change becomes ~2,700 document rewrites, and a reorg day multiplies it by thousands.

+ +
+ +
+
Act 3 · Pattern 1: Extended Reference · B. The model0:37 – 0:38
+

Canonical form vs. the copy: where one edit lands

+

In canonical form the advisor's office is one fact; the relational projection stores it in one row. Extended Reference copies it into every client and account document. Copying the immutable costs nothing later; copying the mutable is write amplification.

+
+
CANONICAL FORM: EACH FACT ONCETHE SAME FACT, EMBEDDEDADVISORPKadvisor_idvarchar2full_namevarchar2officevarchar2desk_phonevarchar2regionvarchar2CLIENTPKclient_idvarchar2full_namevarchar2segmentvarchar2FKadvisor_idvarchar2emailvarchar2ACCOUNTPKaccount_idvarchar2FKclient_idvarchar2acct_typevarchar2balancenumberTRADEPKtrade_idvarchar2FKaccount_idvarchar2symbolvarchar2quantitynumberexec_pricenumberadvisors · A-317office: Leeds → Manchester790client docs1,920account docs≈ 2,700 document rewritesper change · not one transactionRelational projection1 row update · atomicevery consumer sees itmutable reference data: copying it is what creates write amplificationimmutable snapshot: safe to copy
+
+

Snapshot the immutable, reference the mutable. The trade's execution price belongs in the document; the advisor's office does not.

+ +
+ +
+
Act 3 · Pattern 1: Extended Reference · C. Where the knobs flip it0:38 – 0:39
+

The needle flips on reorg day, not on an average Tuesday

+

Two knobs say embed: four consumers and fifty million reads a day. The third knob, where the update lands, says every edit touches ~2,700 documents. Put a number on it and the break-even is about 930 changes a day.

+
+
KNOB SETTINGS · THIS WORKLOAD1 · DiversityHigh: 4 consumersportal, CRM, statements andcopilot all read the card2 · Read / writeExtreme read skew5×10⁷ reads/day against~300 advisor edits/day3 · Update localityRare, fans out ×2,700each edit lands on everydocument that copied itDAILY WORK vs ADVISOR CHANGES / DAY · illustrative modelembedding is cheaperprojection is cheaper50M100M200M500M1B101001k10k100kadvisor changes per day (log)work units per day (log)break-even ≈ 930 / daynormal day · 300reorg day · 8,000Embed the cardProject it · duality view, +15% read +
Both models run on Oracle. The flip point is where the document tax starts.
Illustrative cost model, not a benchmark: 5×10⁷ client reads/day; projecting the card adds ~15% to each read; each document rewrite costs 3 work units; ~2,700 documents embed each advisor.
+
+
+

On a normal day embedding is cheaper: the projection's read cost is real. On reorg day the day's work doubles and it's 22M non-atomic rewrites. Model for the day that hurts.

+ +
+ +
+
Act 3 · Pattern 1: Extended Reference · D. The converged answer0:39 – 0:40
+

Project the mutable: same document, one row to change

+

A duality view serves the document and decides what a write can touch.

+
+
+
+
+

1 · Relational projection: each fact once

+

xr_advisors

+ + + +
advisor_idfull_nameofficedesk
A-317Grace HopperLeedsManchesterx0101
A-402Alan TuringYorkx0144
+

The advisor's office lives in ONE row.

+

xr_clients

+ + + + +
client_idfull_namesegmentprimary_advisor_id
C-104233Ada OkaforAda Okafor-ReidHNWA-317
C-104871Ben AdeyemiMASSA-317
C-105002Cara LindqvistHNWA-402
+
+
+

2 · The duality view: the lab's exact definition

+ 
CREATE JSON RELATIONAL DUALITY VIEW  xr_client_dv AS SELECT JSON {  '_id'      : c.client_id,  'fullName' : c.full_name WITH UPDATE,  'segment'  : c.segment   WITH UPDATE,  'advisor'  : ( SELECT JSON {      'advisorId' : a.advisor_id,      'fullName'  : a.full_name,      'office'    : a.office,      'desk'      : a.desk }    FROM xr_advisors a WITH NOUPDATE    WHERE a.advisor_id = c.primary_advisor_id )} FROM xr_clients c WITH INSERT UPDATE DELETE;
+
    +
  • Can change: fullName, segment; add or delete a client
    • +
  • Can't change: anything under advisor (read-only here)
    • +
+
+
+

3 · Read it: one client document

+ 
{  "_id": "C-104233",  "fullName": "Ada Okafor","Ada Okafor-Reid",  "segment": "HNW",  "advisor": {    "advisorId": "A-317",    "fullName": "Grace Hopper",    "office": "Leeds","Manchester",    "desk": "x0101"  }}
+

C-104871 (also A-317) now reads Manchester too, in the same commit.

+
+
+
+
+ 
SELECT data FROM xr_client_dv WHERE JSON_VALUE(data, '$._id') = 'C-104233';
+

Read: the advisor block is assembled from the one xr_advisors row at read time: a primary-key join on every read. Nothing was copied.

+
+
+ 
UPDATE xr_client_dv SET data = JSON_TRANSFORM(data, SET '$.fullName' = 'Ada Okafor-Reid')
+ WHERE JSON_VALUE(data, '$._id') = 'C-104233';
+

✓ 1 row updated: the xr_clients row for C-104233. No other row is touched.

+
+
+ 
UPDATE xr_client_dv SET data = JSON_TRANSFORM(data, SET '$.advisor.office' = 'Manchester')
+ WHERE JSON_VALUE(data, '$._id') = 'C-104233';
+

ORA-40940: Cannot update field 'office' corresponding to column 'OFFICE' of table 'XR_ADVISORS' in JSON Relational Duality View 'XR_CLIENT_DV': Missing UPDATE annotation or NOUPDATE annotation specified.

+

✗ Rejected, whole statement. The advisor is read-only in this view; its office changes only in its own row.

+
+
+ 
UPDATE xr_advisors SET office = 'Manchester' WHERE advisor_id = 'A-317';
+

✓ 1 row · 1 transaction. Every A-317 client document now shows Manchester at the same commit, with no eventual consistency.

+
+
+
~2,700document rewrites per office move, embedded
+
from eventual consistency
to atomic
+
row, one transaction, duality view1
+
+
+
+
+ +
+ + +
+
Act 3 · Pattern 2: Computed · A. The problem0:40 – 0:42
+

Telecom: a counter on the hottest document you own

+

The app and the charging system both read the balance. To make that one read, each subscriber document carries a running cycle-usage rollup: the Computed pattern. Then 4 billion CDRs a month arrive, and every one rewrites the parent.

+
+
+
+
12M subscribers4B CDRs / month6k/s peak CDRs400+ CDRs/day, heavy user
+
Access patternWhoRateType
Record a CDRmediation1.5k/s avgwrite · hot doc
Show balance in the appself-care app~25M / dayread
Real-time balance checkcharging~4k/sread · <50 ms
"Top 20 talkers this cycle"ops dashboardevery loadread · sort 12M
Bill runbatch12M / cycleread · batch
+
+
subscribers · one document per subscriber{"_id": "S-4471902","msisdn": "+1-202-555-0142","plan": "UNLIMITED-5G","addOns": [ "roam-EU", "hotspot-50" ],"devices": [ { "imei": "35-2099", ... } ],"cycle": "2026-09","cycleUsage": {"totalMB": 48213.7,"totalMin": 1284,"cost": 86.42,"cdrCount": 11906},"updatedAt": "2026-09-28T14:02:11Z"}every CDR rewritesthe whole document:400+ times a dayTop-N sorts 12Mdocuments by thisburied field
+
+
+

The rollup makes the balance read one document. Every CDR pays for it with a whole-document rewrite, and the busiest subscribers take the most rewrites.

+ +
+ +
+
Act 3 · Pattern 2: Computed · B. The model0:42 – 0:43
+

Facts vs. the rollup: where every CDR lands

+

In canonical form a CDR is an immutable fact, appended once, and the cycle rollup is one derived value in one narrow row. Embedded, the rollup lives in the subscriber document, so every CDR rewrites it; sharding the counter moves the pain to the read.

+
+
CANONICAL FORM: FACTS APPENDED, ROLLUP DERIVEDTHE SAME ROLLUP, EMBEDDEDSUBSCRIBERPKsubscriber_idvarchar2msisdnvarchar2planvarchar2cyclevarchar2CDR · append-onlyPKcdr_idnumberFKsubscriber_idvarchar2mbnumberminutesnumbercostnumbercdr_tstimestampSUBSCRIBER_USAGEPKsubscriber_idvarchar2total_mbnumbertotal_minnumbercostnumberAFTER INSERT trigger+1 counter row, same txnCDR stream1,500/s avg · 6k/s peakS-4471902 · one hot document400+ whole-doc rewrites / dayshard 1shard 2shard 3shard Nbalance read = N reads + SUMa write problem traded for a read oneRelational projectionappend 1 row + bump 1counter row · atomicmutable rollup: every CDR writes itimmutable CDR facts: append, never rewrite
+
+

Append the facts, derive the rollup. The CDR belongs in an append-only table; the running total belongs in a row the engine maintains, not inside the document it rewrites.

+ +
+ +
+
Act 3 · Pattern 2: Computed · C. Where the knobs flip it0:43 – 0:44
+

The needle flips on the heavy tail, not the average

+

Diversity and reads argue for a precomputed rollup. The third knob, every write lands on one parent document, argues against. Put a number on it: embedding wins below about 4 CDRs a subscriber a day; the heavy user at 400 costs ~2.5× the work.

+
+
KNOB SETTINGS · THIS WORKLOAD1 · DiversityMedium-high: 4 consumersapp, charging, ops dashboardand the bill run2 · Read / writeWrite-heavy on the rollup~30 balance reads/day vs11 CDRs avg · 400+ heavy3 · Update localityEvery write, one parenteach CDR lands on the samehot subscriber documentWORK / SUBSCRIBER / DAY vs CDRs / DAY · illustrative modelembed winsappend + summary is cheaper1003001k3k10k1101001kCDRs per subscriber per day (log)work units / subscriber / day (log)break-even ≈ 4.4 / daylight · 1.5average · 11heavy · 400Embedded rollup · doc rewriteAppend + summary · +30% read +
Illustrative cost model, not a benchmark: 30 balance reads/subscriber/day; a CDR rewrites the ~4 KB document (4 units + contention) vs a row insert + counter-row update (2 units). Both serialise per subscriber; only the embedded lock is held for a whole-document rewrite. The projected read costs ~30% more.
+
+
+

For light subscribers the embedded rollup is cheaper: the projection's read premium is real. For the heavy tail it's ~2.5× the work, concentrated on the hottest documents.

+ +
+ +
+
Act 3 · Pattern 2: Computed · D. The converged answer0:44 – 0:45
+

Append the CDR, maintain the rollup: staleness zero

+

Store each CDR as an append-only row; a trigger keeps a one-row summary current in the same transaction. A duality view serves the document.

+
+
+
+
RELATIONAL PROJECTIONMAINTAINED · PROJECTEDCONSUMERScdrappend-only rowssubscriber_usagethe rollup · 1 rowsubscribersmsisdn · planix_usage_topntotal_mb DESCsubscriber_dvduality viewcycleUsage NOUPDATEBill runSQL over cdrOps Top-Nindex range scanSelf-care appMongo API find()ChargingSQL/JSONtriggersame txn
+
When to flip back: a rarely written total (light subscribers, a closed cycle's final bill) can live on the document.
+
+
+ 
-- 1 · each CDR is a small append; the trigger bumps one row
+CREATE TRIGGER trg_usage AFTER INSERT ON cdr FOR EACH ROW BEGIN
+  MERGE INTO subscriber_usage u
+  USING (SELECT :NEW.subscriber_id sid FROM dual) s ON (u.subscriber_id = s.sid)
+  WHEN MATCHED THEN UPDATE SET u.total_mb = u.total_mb + :NEW.mb, ...
+  WHEN NOT MATCHED THEN INSERT ... ; END;
+-- 2 · Top-N is an index range scan, not a sort
+CREATE INDEX ix_usage_topn ON subscriber_usage (total_mb DESC, subscriber_id);
+SELECT subscriber_id, total_mb FROM subscriber_usage ORDER BY total_mb DESC
+FETCH FIRST 20 ROWS ONLY;
+
Embedded rollupAppend + summary
Record a CDRwhole-doc rewrite · 51,956 B redo (1,000-CDR doc)insert + counter row · 1,592 B (33×)
Balance read1 document1 document, PK join (~+30%)
Top 20 talkerssort 12M docs · 60 sindex scan · 500 ms
+
Same security model: duality views and projections go through the same privileges, auditing and policies as the tables underneath. No second security model to manage.
+
+
+
+

The rollup is still precomputed and still current. The write went from a whole-document rewrite to two small rows, and Top-N from 60 s to 500 ms (anonymized field result), for a modest, predictable read cost.

+ +
+ + +
+
Act 3 · Pattern 3: Bucket · A. The problem0:45 – 0:47
+

Manufacturing IoT: one bucket, 36,000 rewrites

+

18,000 sensors, 40 plants, ~34k readings a second. To avoid billions of tiny documents the team buckets each sensor's hour into one document: an array plus running counters. The dashboard reads one document, and every incoming reading rewrites it.

+
+
+
+
18,000 sensors40 plants~34k/s readings36k per hot bucket
+
Access patternWhoRateType
Ingest a readinggateways~34k/swrite · hot doc
Last-hour machine dashboardline supervisors6/minread · 1 bucket
Shift rollup (8 h)plant managers120/dayread · range
7-day anomaly scan, all sensorsreliability eng.hourlyread · times out
ML feature extractiondata sciencenightlyread · scan
+
+
readings · one bucket per sensor per hour{"_id": "M-118|VIB|2026-09-25T14","machineId": "M-118","metric": "VIB","hourStart": "2026-09-25T14:00:00Z","count": 21604,"sum": 1548212.7,"min": 0.8, "max": 97.3,"readings": [{ "ts": "14:00:00.1", "val": 71.4 },{ "ts": "14:00:00.2", "val": 71.9 },... 21,601 more readings ...{ "ts": "14:36:00.4", "val": 72.2 }]}counters: $inc / $maxrewritten on everysingle reading$push lands here:rewrites 2.6 MBand it only grows
+
+
+

The bucket buys a one-document dashboard read. It pays with a write whose cost grows with every reading, and the hottest sensors grow fastest.

+ +
+ +
+
Act 3 · Pattern 3: Bucket · B. The model0:47 – 0:48
+

Immutable readings, mutable bucket

+

In canonical form a reading is an immutable fact; in the relational projection it is one row, inserted once. The hourly counters are derived. The bucket turns those immutable facts into one ever-growing document that every reading rewrites.

+
+
CANONICAL FORM: EACH READING ONCE, NEVER REWRITTENONE HOT SENSOR-HOUR, BUCKETEDPLANTPKplant_idvarchar2namevarchar2regionvarchar2MACHINEPKmachine_idvarchar2FKplant_idvarchar2modelvarchar2line_nonumberSENSOR_READINGSPKreading_idnumberFKmachine_idvarchar2metricvarchar2reading_valnumberreading_tstimestampHOURLY_MV · derivedmachine_idvarchar2hour_startdatennumbersum_valnumbermax_valnumberGROUP BY hour · on readM-118 · spindle vibration10 readings / s1 bucket documentcount·sum·min·maxreadings[ ]every reading rewrites bothO(n²) rewrite volume36,000 rewrites · 4.3 MB bucket≈ 78 GB if each push rewrites itRelational projection36,000 row inserts · ≈ 9 MBappended to one partitionKey:red: rewritten on every reading (the bucket's counters and array)green: immutable (SENSOR_READINGS rows, inserted once)
+
+

Immutable facts, mutable container. The write amplification comes from the bucket, not from the data. Derive the counters instead of storing them.

+ +
+ +
+
Act 3 · Pattern 3: Bucket · C. Where the knobs flip it0:48 – 0:49
+

Slow sensors love buckets. Hot sensors break them.

+

Diversity is medium and the workload is write-heavy, but the knob that bites is locality: every reading lands on the same growing document. Rewrite cost grows with n², so the break-even is about 900 readings per bucket.

+
+
KNOB SETTINGS · THIS WORKLOAD1 · DiversityMedium: 4 read shapesdashboard, shift rollup,anomaly scan, ML features2 · Read / writeWrite-heavy~3B readings in per day;reads are rollups, not points3 · Update localityEvery write, same documenteach reading rewrites thegrowing hour bucketBYTES PER SENSOR-HOUR vs READINGS / BUCKET · illustrative modelbucket is cheaperrows are cheaper100 KB10 MB1 GB100 GB1001k10k100kreadings per bucket (log)bytes touched per sensor-hour (log)break-even ≈ 9001 / min · 601 / s · 3,600hot · 36,000Bucket · $push + countersRows · append, GROUP BY on read16 MB cap ≈ 140k +
Illustrative cost model, not a benchmark: 120 B per reading; each $push rewrites the whole bucket (s·n²/2 per hour); a row insert adds ~130 B of header, index and redo; a live dashboard reads the hour 360×/h, paying ~150 B more per row than one 2 KB-overhead document fetch.
+
+
+

Below ~900 readings the one-document read wins, so bucket the slow sensors. At 1/s the bucket costs ~2.7× the rows; the hot spindle ~22×, marching at the 16 MB cap.

+ +
+ +
+
Act 3 · Pattern 3: Bucket · D. The converged answer0:49 – 0:50
+

Let the partition be the bucket, and put SQL in the pipeline

+

Store each reading as an append-only row in an hourly INTERVAL partition. The document team keeps its pipeline and hands the heavy rollup to SQL.

+
+
+
+
RELATIONAL PROJECTIONSUMMARYCONSUMERSmachinesplant · line · modelsensor_readingsappend-only rowsINTERVAL 1 hGROUP BY hourpartition-prunedparallelhourly_mvsummary MVDashboardsSQL · last hourAnomaly scanMongo API $sqlShift reportSQL · MV
+
When to flip back: slow sensors. At one reading a minute a bucket holds 60 readings, and below ~900 readings per bucket the one-document read wins.
+
+
+
$sql: Oracle's addition to the MongoDB aggregation pipeline. It runs full SQL as one stage of db.aggregate().
+ 
CREATE TABLE sensor_readings ( machine_id VARCHAR2(16), metric VARCHAR2(24),
+  reading_val NUMBER(12,3), reading_ts TIMESTAMP )
+PARTITION BY RANGE (reading_ts) INTERVAL (INTERVAL '1' HOUR)
+( PARTITION p0 VALUES LESS THAN (TIMESTAMP '2026-08-01 00:00:00') );
+
+db.aggregate([{ $sql: `
+  select json { 'machineId' : machine_id, 'n' : count(*), 'max' : max(reading_val),
+                'hour' : to_char(trunc(reading_ts,'HH24'),'YYYY-MM-DD"T"HH24') }
+  from sensor_readings where reading_ts >= systimestamp - interval '7' day
+  group by machine_id, metric, trunc(reading_ts,'HH24')` }])
+
Hour bucketPartitioned rows
Ingest, hot sensor at :59rewrite a 4.3 MB docinsert one ~120 B row
Last-hour dashboard1 document fetchpruned range scan (~2× read bytes)
Bucket rollup · anonymized field result29 s< 400 ms
+
+
+
+

Ingest becomes a constant-cost insert with no 16 MB ceiling. An anonymized field reshape took a 29 s bucket rollup to under 400 ms, paid for with a heavier last-hour read.

+ +
+ + +
+
Act 3 · Pattern 4: Subset · A. The problem0:50 – 0:52
+

Insurance: ten claim events inline, rewritten on every event

+

Agents and the customer app open the policy summary 2M times a day. To make it one read, the team kept the 10 most recent claim events inline and the full history in a second collection: the Subset pattern. Now watch the write path.

+
+
+
+
8M policies10 events inline30k events / day4 consumers
+
Access patternWhoRateType
Record a claim eventadjusters · ops~30k / daywrite · hot doc
Policy summary pageagents · app~2M / dayread
Full claim historyadjusters · audit~5k / dayread · 2nd query
Fraud review across claimsSIU analystsnightlyread · cross-doc
Catastrophe (CAT) surgeclaims ops~300k / daywrite · hot doc
+
+
policies · one document per policy{"_id": "POL-7730142","holder": "R. Alvarez","product": "HOME","coverages": [ ... ],"recentEvents": [{ "claim": "CLM-5581", "type": "PAYMENT","amount": 2400, "ts": "09-27T14:02" },{ "claim": "CLM-5581", "type": "NOTE" },... 8 more · sorted · trimmed to 10]}rewritten on every event$push · $sort · $slice: 10~30k times a dayclaim_history · one document per event{ "_id": "EV-90311", "policy": "POL-7730142","claim": "CLM-5581", "type": "PAYMENT" }…and a second writesame event, another placeto keep in step
+
+
+

The summary read is why the pattern exists. Every claim event pays for it twice: a push-sort-trim rewrite of the policy document plus a history insert, in two places that must agree.

+ +
+ +
+
Act 3 · Pattern 4: Subset · B. The model0:52 – 0:53
+

An append-only event, turned into a document rewrite

+

Canonically a claim event is an immutable fact, inserted once. The Subset pattern turns it into two writes: a push-sort-trim rewrite of the policy document and a history insert, to maintain a list an index could simply read.

+
+
CANONICAL FORM: EACH EVENT ONCEONE CLAIM EVENT, TWO MODELSPOLICYPKpolicy_idvarchar2holdervarchar2productvarchar2premiumnumberstatusvarchar2CLAIMPKclaim_idvarchar2FKpolicy_idvarchar2loss_datedatestatusvarchar2reservenumberCLAIM_EVENT · append-onlyPKevent_idvarchar2FKclaim_idvarchar2FKpolicy_idvarchar2event_typevarchar2amountnumberevent_tstimestampINDEXpolicy_id, event_ts DESCclaim event · CLM-5581payment · $2,400policy docpush · sort · trimwhole doc rewrittenhistory docinsert full eventsecond place2 writes, 2 places, per eventthe hot doc absorbs every oneRelational projection1 row insert + 1 index entrythe recent 10 is derived on read,not maintainedmutable claim state: updated in placeimmutable event facts: append once, never rewritten
+
+

The recent ten is a query, not a fact. Store the event once; a (policy_id, event_ts DESC) index keeps it pre-sorted. The price is one index entry per insert and a short range scan per read.

+ +
+ +
+
Act 3 · Pattern 4: Subset · C. Where the knobs flip it0:53 – 0:54
+

The Subset pattern wins on sunny days, not in storms

+

Two knobs favour the Subset pattern: four consumers and a 67 : 1 read skew. The third (every event rewrites the parent) sets the price. Break-even is about 83k events a day; a CAT day runs at 300k.

+
+
KNOB SETTINGS · THIS WORKLOAD1 · DiversityHigh: 4 consumersagents, customer app, adjustersand fraud all read claim events2 · Read / writeRead-heavy, ~67 : 12M summary reads/day against~30k claim events/day3 · Update localityEvery event hits the parenteach event rewrites the policydoc and writes history tooDAILY WORK vs CLAIM EVENTS / DAY · illustrative modelthe Subset pattern is cheaperquerying is cheaper3M5M10M10k100k1Mclaim events per day (log)work units per day (log)break-even ≈ 83k / daynormal day · 30kCAT day · 300kSubset · push + trim + historyQuery it · FETCH FIRST 10, +25% read +
Illustrative cost model, not a benchmark: 2×10⁶ summary reads/day; assembling the summary on read (policy row + index range scan, FETCH FIRST 10) adds ~25% per read; a policy-document rewrite costs 6 work units, a row or document insert 2; full-history reads cost the same in both models.
+
+
+

At 30k events a day the Subset pattern is cheaper: the query's read premium is real. On a CAT day it costs ~40% more, all of it on the hottest documents.

+ +
+ +
+
Act 3 · Pattern 4: Subset · D. The converged answer0:54 – 0:55
+

Stop maintaining the subset: insert once, read the top ten

+

Store every claim event once in one table. A (policy_id, event_ts DESC) index makes the recent ten a range scan that stops after ten rows, and SQL/JSON assembles the same policy document on read. Hot and cold are one table.

+
+
+
+
RELATIONAL PROJECTIONSAME ROWS, TWO READSCONSUMERSpoliciesheader, onceclaim_eventsappend · DESC indexclaimsstatus · reservepolicy summarySQL/JSONFETCH FIRST 10full historysame query,no FETCH FIRSTAgents · appSQL/JSON docAdjustersSQL · auditFraud reviewSQL analytics
+
When to flip back: if the inline slice is frozen at write time and rarely changes (the three coverages printed on the declarations page), keep it in the document. A slice that changes on every event is a query.
+
+
+

Stop maintaining the subset: query it

CREATE INDEX ce_recent ON claim_events (policy_id, event_ts DESC);
+INSERT INTO claim_events VALUES (:ev, :clm, :pol, 'PAYMENT', 2400, SYSTIMESTAMP);
+-- ↑ the whole write.  ↓ the policy summary, assembled on read
+SELECT JSON_OBJECT('policyId' VALUE p.policy_id,
+  'recentEvents' VALUE (SELECT JSON_ARRAYAGG(
+      JSON_OBJECT('type' VALUE event_type, 'amount' VALUE amount)
+      ORDER BY event_ts DESC)
+    FROM (SELECT * FROM claim_events e WHERE e.policy_id = p.policy_id
+          ORDER BY e.event_ts DESC FETCH FIRST 10 ROWS ONLY)))
+FROM policies p WHERE p.policy_id = :pol;
+
SubsetQuery it
Claim eventdoc rewrite + history insert1 row + 1 index entry
Policy summary read1 document fetchindex range scan, 10 rows (~+25%)
CAT day · 300k events300k doc rewrites300k inserts
+
+
+
+

The app gets the same policy document. Each claim event went from a document rewrite plus a history insert to one row, paid for with a modest, predictable read cost.

+ +
+ + +
+
Act 3 · Pattern 5: Tree / Hierarchy · A. The problem0:55 – 0:57
+

Manufacturing BOM: the path is fast until engineering moves it

+

An equipment maker stores 1.2M parts with a materialized path, so exploding any assembly is one prefix scan. Then ~200 change orders a day re-parent subassemblies, and quality asks the reverse question: which products contain part X?

+
+
+
+
1.2M parts12 levels deep~200 ECOs / day10–40k parts per move
+
Access patternWhoRateType
Explode an assemblyMRP · service~2M/dayread
Cost rollup up the treecostingnightlyread
Where-used: which products contain X?sourcing~5k/dayread · reverse
Recall impact analysisqualityburstsread · reverse
ECO: re-parent a subassemblyengineering~200/daywrite · fan-out
+
+
parts · one document per part{"_id": "P-88213","name": "pump seal kit","rev": "C","unitCost": 14.20,"path": "root.A-1.SA-7.HP-3","depth": 4,"usedIn": ["HP-3", "HP-9", "VB-12"],"rollupCost": 14.20,"supplier": "S-417"}path = positionan ECO above it rewritesthis and every descendantsecond structure forwhere-used: parent list,kept in sync by app code
+
+
+

The path makes the downward read one index scan. It makes every re-parent a subtree-sized rewrite, and it can't answer where-used without a second, hand-synced structure.

+ +
+ +
+
Act 3 · Pattern 5: Tree / Hierarchy · B. The model0:57 – 0:58
+

Store the edge, derive the position

+

In canonical form each parent → child link is one relationship, stored as one row in the relational projection; position is computed by walking edges. The materialized path stores position in every part, so moving one subassembly rewrites the whole subtree beneath it.

+
+
CANONICAL FORM: EDGES, NOT BAKED PATHSTHE SAME MOVE, AS PATHSPARTPKpart_idvarchar2namevarchar2revvarchar2unit_costnumberFKsupplier_idvarchar2BOM_EDGEPKedge_idnumberparent_idvarchar2FKchild_idvarchar2qtynumberECOPKeco_idvarchar2FKedge_idnumberfrom_parentvarchar2released_attimestampparent → child · a part can have many parents (DAG)ECO-4471 · re-parent SA-7A-1 → A-21SA-7 itself10 – 40kdescendants≈ subtree-size rewritesper ECO · not one transactionAdjacency edge: 1 row updatethe subtree follows · atomicthe column an ECO writes: structure, not positionimmutable change history: safe to copy
+
+

A path is a derived fact stored N times. The edge is the fact stored once: an ECO changes one row and the subtree follows.

+ +
+ +
+
Act 3 · Pattern 5: Tree / Hierarchy · C. Where the knobs flip it0:58 – 0:59
+

The needle flips on the size of the move

+

Two knobs favour the path: four read questions and a read-heavy day. The third (every re-parent lands on the whole subtree) decides it. Charge the recursive read its real premium and break-even sits near 1,000 parts per move.

+
+
KNOB SETTINGS · THIS WORKLOAD1 · DiversityHigh: 4 questionsexplode, roll up, where-used,recall impact2 · Read / writeRead-heavy~2M explosions/day against~200 ECOs/day3 · Update localityLands on every descendantone re-parent rewrites10 to 40,000 part pathsDAILY WORK vs PARTS PER MOVED SUBTREE · illustrative modelpath is cheaperedges are cheaper2M5M10M20M101001k10kparts in the moved subtree (log)work units per day (log)break-even ≈ 1,000 partsroutine ECO · 50module move · 10kMaterialized pathAdjacency edges · CONNECT BY, +30% read +
Illustrative cost model, not a benchmark: 2×10⁶ explosion reads/day at 1 unit via a path-prefix scan; recursive CONNECT BY adds ~30% per explosion; 200 ECOs/day; each document or row rewrite costs 3 units. Where-used is left out, which favours the path.
+
+
+

For small moves and static trees the path is cheaper: the recursive read costs more. Once engineering moves modules of thousands of parts, the edge wins: ~3× at 10k, ~10× at 40k.

+ +
+ +
+
Act 3 · Pattern 5: Tree / Hierarchy · D. The converged answer0:59 – 1:00
+

Store the edge: recurse down, match up, move one row

+

Keep the BOM as parent → child edges: CONNECT BY explodes it, a SQL/PGQ graph over the same rows answers where-used, and an ECO re-parent is one row.

+
+
+
+
RELATIONAL PROJECTIONSAME ROWS, TWO READSCONSUMERSbom_edgesparent → childECO writes hereparts1.2M rowsCONNECT BYexplode · roll upone recursive passbom_graphSQL/PGQ graphwhere-used, N hopsPlanningSQL · explodeCost rollupSQL · nightlyRecall deskwhere-usedSQL/PGQ
+
When to flip back: a tree that rarely moves and is read one way (a product catalogue, an org chart snapshot), keep the materialized path. Its prefix scan is the cheapest explosion there is.
+
+
+

Store the edge: recurse down, match up, move with one row

-- explode: everything under an assembly, one pass
+SELECT child_id, qty, LEVEL FROM bom_edges
+START WITH parent_id = :asm CONNECT BY PRIOR child_id = parent_id;
+
+-- where-used: the reverse question, same rows (bom_graph)
+SELECT assembly FROM GRAPH_TABLE ( bom_graph
+  MATCH (a IS part)-[IS contains]->{1,4}(p IS part)
+  WHERE p.part_id = :part COLUMNS (a.part_id AS assembly) );
+
+-- ECO-4471: re-parent SA-7 = ONE row, subtree follows
+UPDATE bom_edges SET parent_id = 'A-2'
+WHERE  parent_id = 'A-1' AND child_id = 'SA-7';
+
Materialized pathEdges + graph
ECO re-parent1 doc per part in subtree1 edge row
Explode an assembly1 prefix range scanCONNECT BY pass (~+30%)
Where-used2nd structure, app-syncedGRAPH_TABLE, same rows
+
+
+
+

The re-parent went from one rewrite per descendant to one edge row, and where-used went from a hand-synced second structure to a graph match, paid for with a real recursive-read premium.

+ +
+ + +
+
Act 3 · Pattern 6: Outlier · A. The problem1:00 – 1:02
+

Brokerage: 40,000 books of business, and 120 that don't fit

+

Every advisor's “my book” screen is one document with the client list embedded. For the median book of 180 clients that is perfect. Then the team books with 150,000 clients hit the 16 MB cap, and the Outlier pattern adds a flag, overflow documents and a branch.

+
+
+
+
40,000 advisors~17M clientsp50 180 clients~120 books > 20k
+
Access patternWhoRateType
My book: top 100 by AUMadvisor portal~2k/s at the openread
Add a clientonboarding~30k / daywrite
Nightly AUM refreshbatch~17M clients / nightwrite · big doc
Advisor transition: move a bookops~50 books / weekwrite · big doc
+
+
advisors · one document per book{"_id": "A-900","name": "Institutional Desk 4","bookType": "team","hasExtras": true,"clientCount": 148212,"clients": [{ "id": "C-900001","name": "Harbor Pension Trust","aum": 912400000 },{ "id": "C-900002","aum": 887150000 },... 19,998 more ...]}the 16 MB cap, leaked:128k more clients sitin 7 overflow docsnightly AUM refreshrewrites the whole~8 MB document
+
+
+

The pattern is built for the median book. The 16 MB cap becomes a flag, seven extra documents and an if in every reader, and the biggest documents take the most writes.

+ +
+ +
+
Act 3 · Pattern 6: Outlier · B. The model1:02 – 1:03
+

The book is rows; the document boundary is the outlier

+

A client is its own entity, stored once, as a row keyed by advisor: 180 rows or 148,000, same table, same index. The document model makes the book one physical object, so the fat tail becomes overflow documents and an application branch.

+
+
CANONICAL FORM: A CLIENT IS ITS OWN ENTITYTHE SAME BOOK, AS DOCUMENTSDESKPKdesk_idvarchar2desk_namevarchar2book_typevarchar2ADVISORPKadvisor_idvarchar2FKdesk_idvarchar2full_namevarchar2regionvarchar2CLIENTPKclient_idvarchar2FKadvisor_idvarchar2full_namevarchar2aumnumberaum_as_ofdateopened_ondateINDEX (advisor_id, aum DESC)any book, any size: one range scanp50 advisor · 180 clients1 document · ~72 KB · one readteam book · 148k clientsadvisor doc: first 20k + hasExtras#part-220k clients#part-320k clients… #part-820k clientsapplication branchif hasExtras: 1 + 7 fetches, merge-sortRelational projection1 query, any booksame index range scan · no branchmutable: rewritten nightly, inside the biggest documentsimmutable: safe to copy anywhere
+
+

Outliers exist only where a physical boundary is part of the model. Remove the boundary and a 148k-client book is just a longer range scan.

+ +
+ +
+
Act 3 · Pattern 6: Outlier · C. Where the knobs flip it1:03 – 1:04
+

Designed for the p50, paid for at the p99.9

+

For the median book of 180 clients, one embedded document is ~3× cheaper than a range scan; the break-even is about 3,200 clients. The ~120 team books sit 50× past it, where the read becomes 1 + k fetches and a merge in the app.

+
+
KNOB SETTINGS · THIS WORKLOAD1 · DiversityLow: one book screenportal and batch read thebook the same way2 · Read / writeRead-heavy by day~2k reads/s at the open;one refresh storm a night3 · Update localitySkew ×800, p50 → maxwrites land in the biggestdocuments on the platformWORK PER TOP-100 BOOK READ vs BOOK SIZE · illustrative modelone document is cheaperrange scan is cheaper10301003001k3k101001k10k100kclients per advisor (log)work units per read (log)break-even ≈ 3,200 clientsp50 · 180 clientsteam book · 148kEmbedded book + overflowRows · index range scan, top 100 +
Illustrative cost model, not a benchmark: embedded = 10 per document fetch + 0.01 per client read (~400 B each), 20k-client overflow documents + 0.005/client app merge; rows = 10 + 2·log₁₀ n + 0.25 per row returned (≤ 100).
+
+
+

The embedded book wins for 99% of advisors. Say so. At 148k clients it's ~50× the work and a second code path that only the 120 largest, most-written books exercise.

+ +
+ +
+
Act 3 · Pattern 6: Outlier · D. The converged answer1:04 – 1:05
+

The outlier is just more rows: one query, no branch

+

Store each client as a row, index it by (advisor_id, aum DESC), and project the book as a document with a duality view where a consumer wants one. The book screen pages with FETCH FIRST / OFFSET: the same statement for every advisor.

+
+
+
+
RELATIONAL PROJECTIONSAME ROWS, TWO READSCONSUMERSadvisorsone row eachclientsidx advisor_id,aum DESCadvisor_dvduality viewbook as a documenttop-N queryFETCH FIRST ·OFFSET pagesCRMMongo API find()AI assistantSQL/JSONBook screenSQL · any size
+
When to flip back: if the list has a hard upper bound (ten saved watchlists, five beneficiaries), embed it. The Outlier pattern is only needed when the bound is a guess. Growth costs too: +1 client to an 800-client book is ~20× a row insert's redo (companion lab).
+
+
+

One table, one index, one query, no overflow branch

CREATE JSON RELATIONAL DUALITY VIEW advisor_dv AS
+SELECT JSON { '_id'     : a.advisor_id,
+  'clients' : [ SELECT JSON { 'clientId' : c.client_id,
+                                'aum'      : c.aum WITH UPDATE }
+                FROM clients c WITH INSERT UPDATE DELETE
+                WHERE c.advisor_id = a.advisor_id ]
+} FROM advisors a WITH INSERT UPDATE DELETE;
+-- book screen: index (advisor_id, aum DESC), any book size
+SELECT client_id, aum FROM clients WHERE advisor_id = :advisor
+ORDER BY aum DESC
+OFFSET :skip ROWS FETCH NEXT 100 ROWS ONLY;
+
Outlier patternRows + projection
Top 100, 148k book8 fetches + app merge1 range scan, 100 rows
Top 100, 180-client book1 document (cheaper)range scan + assembly (~3× in model)
Move a bookrewrite 2 books, may overflow1 UPDATE, 1 transaction
+
+
+
+

The 16 MB cap never enters the model because the book was never one document. One code path for 40,000 advisors, paid for with a modest, predictable read cost at the median.

+ +
+ + + +
+
Act 3 · The rest of the catalog1:05 – 1:09
+

The rest of the catalog, at speed

+
    +
  • Still useful, now optional: the engine already does the work
    • +
  • Obsolete: the limit they fought is gone
    • +
+
+
STILL USEFUL, NOW OPTIONALthe engine already gives you the mechanismPolymorphicmixed shapes in one collectionmultivalue indexAttributesparse key/value attributesreal index on key/valueSchema-Versioningold and new shapes side by sideIS JSON + a viewDocument-Versioninghistory of every changetemporal validity / FlashbackOBSOLETEthe limit they fought is goneApproximationhand-rolled approximate countersAPPROX_COUNT_DISTINCTPre-Allocationpadding so a document won't movean MMAPv1 artifactPatterns are bets against an engine.
+
+ +
+ + +
+
Act 3 · Where documents win1:09 – 1:13
+

Where the document model still wins

+
    +
  • The white-hot cart is a JSON COLLECTION TABLE in Oracle, over the MongoDB API or SQL/JSON
    • +
  • Its knobs say embed: one owner, hot reads and writes, private updates
    • +
  • A duality view would add k index probes to every read; joins are never free
    • +
+
+
THE CART, IN ORACLECREATE JSON COLLECTION TABLE carts;{ "_id": "cart#u42","items": [{ "sku": "A-19", "qty": 2 },{ "sku": "C-07", "qty": 1 } ] }one table, two access surfacesMongoDB APIinsertOne() · find()SQL/JSONJSON_VALUE · JSON_TABLETHE KNOBS FOR THE CART1 · DiversityLow: one ownerone shopper reads andwrites this cart2 · Read / writeBoth hotevery click reads itand writes it3 · Update localityPrivatean update touches one cart,never a shared factembed wins: nothing shared to copyREAD COST, SAME DOCUMENT SHAPEJSON collectionO(log n)1 probe + 1 fetchDuality view over k tables… kassembleO(k · log n)k probes + assemblynested arrays cost morejoins are never freePowerful tools: pick the shape the workload asks for. A single collection is not a duality view.
+
+ +
+ + +
+
Act 4 · How you know where the needle flips1:13 – 1:17
+

Don't argue the model. Bake it off.

+
    +
  • Your schema, your data, five candidate shapes, one engine
    • +
+
+
1Your schemathe real one,not a toy2Your dataat productioncardinality3?D0as-isD1normalizedD2embedded{ }D3hybridD4dualityCandidate designsD0–D4 · same data, same engine4readswritesMeasurereads AND writes,at your write rate5The crossoverwhere the winnerflipsRULE 1Writes are access patterns tooa shape that wins reads and loses writes hasn't wonRULE 2A win at the wrong scale is not a winmeasure at production cardinality, or you're tuning a toyReason to the candidates. Measure to the decision.
+
+ +
+ + +
+
Act 4 · How you know where the needle flips1:17 – 1:20
+

Two engagements where the preferred design lost to the measurement

+
    +
  • Both anonymized, both measured at production cardinality
    • +
+
+
GLOBAL BANKHeavy aggregation query30 s1.0 sbeforeafter28× fasterQuery rewrite only.No new model.GLOBAL BANKElected data model30 s18 msbeforeafter1,667× fasterA hybrid model won:relational tables + JSON columns.WEALTH-MANAGEMENT PLATFORM40 production queries0.83 s0.16 sbeforeafter−81% on average40 of 40 faster.A query that timed out: 0.39 s.
+
+ +
+ + +
+
Act 4 · How you know where the needle flips1:20 – 1:23
+

And sometimes the document shape wins, on the same engine

+
    +
  • Measurement cuts both ways: the method isn't rigged toward relational
    • +
  • One shape per access pattern, all on one engine
    • +
+
+
ENTITLEMENTS · ARRAY CONTAINMENT“which resources can this principal touch?”27 tablesa join thicket{ "principal": "p-88","resources": ["r-12","r-40", "r-77", ...] }multivalue indexone document, indexed array~9–15× faster on the same engineTHE 90/10every query on the shape that suits it~90%: happy on a document shape10%~10%: joins, aggregation, graphone canonical form · one engine
+
+ +
+ + +
+
Act 5 · Close1:23 – 1:26
+

Five tests for a converged engine

+

"Multi-model" is a marketing word: most products use several engines behind one logo. Converged is a physical claim.

+
+
1One transactionboundaryone ACID commitspans every model2One optimizerone cost-based planacross every model3One consistencymodelno lag between avector and its row4One governancedomainone auth, one audit,one patch cycle5Shared accesssurfacesSQL, SQL/JSON, SQL/PGQ,MongoDB API: same rowsThe test: if any answer is “that's a different service,” it's polyglot with a shared bill.ADDED BENEFITOne database to replicate, not five stores to keep in syncMulti-region and multi-cloud get far simpler: one replication stream,one failover plan, one consistency story.“Tell the DBAs your app is ready for multi-region and multi-cloud.”one databaseregion Bcloud C
+
+ +
+ + +
+
Act 5 · Close1:26 – 1:28
+

Five questions for any architecture, including yours

+
    +
  • Take these back to your own stack. If you can't answer one, that's the finding.
    • +
+
+
1Where is your canonical form?2Which projected shapes do your consumers need?3Which access dimensions, and how often?4What is your projection latency?the quiet one5What consistency does your AI need?
+
+ +
+ + +
+
Act 5 · Close1:28 – 1:30
+

Bring your slowest query. We'll measure it.

+
    +
  • You don't have to believe any number in this deck: reproduce it on your workload
    • +
+
+
1Bring your schemaand your five slowest queries2We bake it offon your data, at your cardinality3Numbers pick the modelreads and writes, measuredWorkshop 3 · the lab“One restaurant, every model”: you buildthe projections yourself, hands-on.Companion repo · converged-modeling-patternsEvery pattern in this deck as runnable SQL,validated on Oracle AI Database 26ai Free.Model the domain, not the engine. Project the access.Measure the tradeoff. Don't argue it.
+
+ +
+ +
+
+ CONVERGED DATA MODELING · WORKSHOP 1 + ← → to navigate · Home/End to jump · N for notes + + 1 / 45 +
+ + + +
+
+
+ + + + diff --git a/labs/converged-modeling-patterns/presentations/fonts/LICENSE-bricolage-grotesque.txt b/labs/converged-modeling-patterns/presentations/fonts/LICENSE-bricolage-grotesque.txt new file mode 100644 index 0000000..49135ef --- /dev/null +++ b/labs/converged-modeling-patterns/presentations/fonts/LICENSE-bricolage-grotesque.txt @@ -0,0 +1,93 @@ +Copyright 2022 The Bricolage Grotesque Project Authors (https://github.com/ateliertriay/bricolage) + +This Font Software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +http://scripts.sil.org/OFL + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/labs/converged-modeling-patterns/presentations/fonts/LICENSE-ibm-plex-mono.txt b/labs/converged-modeling-patterns/presentations/fonts/LICENSE-ibm-plex-mono.txt new file mode 100644 index 0000000..83eac82 --- /dev/null +++ b/labs/converged-modeling-patterns/presentations/fonts/LICENSE-ibm-plex-mono.txt @@ -0,0 +1,93 @@ +Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-ThinItalic.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-ExtraLight.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-ExtraLightItalic.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-Light.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-LightItalic.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-Regular.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-Italic.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-Medium.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-MediumItalic.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-SemiBold.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-SemiBoldItalic.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-Bold.ttf: Copyright 2017 IBM Corp. All rights reserved. IBMPlexMono-BoldItalic.ttf: Copyright 2017 IBM Corp. All rights reserved. + +This Font Software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +http://scripts.sil.org/OFL + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/labs/converged-modeling-patterns/presentations/fonts/LICENSE-public-sans.txt b/labs/converged-modeling-patterns/presentations/fonts/LICENSE-public-sans.txt new file mode 100644 index 0000000..a6cb63a --- /dev/null +++ b/labs/converged-modeling-patterns/presentations/fonts/LICENSE-public-sans.txt @@ -0,0 +1,93 @@ +Copyright 2015 The Public Sans Project Authors (https://github.com/uswds/public-sans) PublicSans-Italic[wght].ttf: Copyright 2015 The Public Sans Project Authors (https://github.com/uswds/public-sans) + +This Font Software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +http://scripts.sil.org/OFL + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/labs/converged-modeling-patterns/presentations/fonts/bricolage-grotesque-latin-40c2cb34.woff2 b/labs/converged-modeling-patterns/presentations/fonts/bricolage-grotesque-latin-40c2cb34.woff2 new file mode 100644 index 0000000..fcc4eb1 Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/fonts/bricolage-grotesque-latin-40c2cb34.woff2 differ diff --git a/labs/converged-modeling-patterns/presentations/fonts/bricolage-grotesque-latin-ext-fdb6b064.woff2 b/labs/converged-modeling-patterns/presentations/fonts/bricolage-grotesque-latin-ext-fdb6b064.woff2 new file mode 100644 index 0000000..6e8caa8 Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/fonts/bricolage-grotesque-latin-ext-fdb6b064.woff2 differ diff --git a/labs/converged-modeling-patterns/presentations/fonts/fonts.css b/labs/converged-modeling-patterns/presentations/fonts/fonts.css new file mode 100644 index 0000000..c4b8d24 --- /dev/null +++ b/labs/converged-modeling-patterns/presentations/fonts/fonts.css @@ -0,0 +1,187 @@ +/* Google Fonts (SIL OFL 1.1), bundled so the deck works offline. See LICENSE-*.txt. */ +/* latin-ext */ +@font-face { + font-family: 'Bricolage Grotesque'; + font-style: normal; + font-weight: 500; + font-stretch: 100%; + font-display: swap; + src: url(bricolage-grotesque-latin-ext-fdb6b064.woff2) format('woff2'); + unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF; +} +/* latin */ +@font-face { + font-family: 'Bricolage Grotesque'; + font-style: normal; + font-weight: 500; + font-stretch: 100%; + font-display: swap; + src: url(bricolage-grotesque-latin-40c2cb34.woff2) format('woff2'); + unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; +} +/* latin-ext */ +@font-face { + font-family: 'Bricolage Grotesque'; + font-style: normal; + font-weight: 700; + font-stretch: 100%; + font-display: swap; + src: url(bricolage-grotesque-latin-ext-fdb6b064.woff2) format('woff2'); + unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF; +} +/* latin */ +@font-face { + font-family: 'Bricolage Grotesque'; + font-style: normal; + font-weight: 700; + font-stretch: 100%; + font-display: swap; + src: url(bricolage-grotesque-latin-40c2cb34.woff2) format('woff2'); + unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; +} +/* latin-ext */ +@font-face { + font-family: 'Bricolage Grotesque'; + font-style: normal; + font-weight: 800; + font-stretch: 100%; + font-display: swap; + src: url(bricolage-grotesque-latin-ext-fdb6b064.woff2) format('woff2'); + unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF; +} +/* latin */ +@font-face { + font-family: 'Bricolage Grotesque'; + font-style: normal; + font-weight: 800; + font-stretch: 100%; + font-display: swap; + src: url(bricolage-grotesque-latin-40c2cb34.woff2) format('woff2'); + unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; +} +/* latin-ext */ +@font-face { + font-family: 'IBM Plex Mono'; + font-style: normal; + font-weight: 400; + font-display: swap; + src: url(ibm-plex-mono-latin-ext-1969f6ff.woff2) format('woff2'); + unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF; +} +/* latin */ +@font-face { + font-family: 'IBM Plex Mono'; + font-style: normal; + font-weight: 400; + font-display: swap; + src: url(ibm-plex-mono-latin-e2661bcc.woff2) format('woff2'); + unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; +} +/* latin-ext */ +@font-face { + font-family: 'IBM Plex Mono'; + font-style: normal; + font-weight: 500; + font-display: swap; + src: url(ibm-plex-mono-latin-ext-93cfae71.woff2) format('woff2'); + unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF; +} +/* latin */ +@font-face { + font-family: 'IBM Plex Mono'; + font-style: normal; + font-weight: 500; + font-display: swap; + src: url(ibm-plex-mono-latin-0bc96cd0.woff2) format('woff2'); + unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; +} +/* latin-ext */ +@font-face { + font-family: 'IBM Plex Mono'; + font-style: normal; + font-weight: 600; + font-display: swap; + src: url(ibm-plex-mono-latin-ext-a7353317.woff2) format('woff2'); + unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF; +} +/* latin */ +@font-face { + font-family: 'IBM Plex Mono'; + font-style: normal; + font-weight: 600; + font-display: swap; + src: url(ibm-plex-mono-latin-1b581b15.woff2) format('woff2'); + unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; +} +/* latin-ext */ +@font-face { + font-family: 'Public Sans'; + font-style: normal; + font-weight: 400; + font-display: swap; + src: url(public-sans-latin-ext-53e02343.woff2) format('woff2'); + unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF; +} +/* latin */ +@font-face { + font-family: 'Public Sans'; + font-style: normal; + font-weight: 400; + font-display: swap; + src: url(public-sans-latin-6e68d19a.woff2) format('woff2'); + unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; +} +/* latin-ext */ +@font-face { + font-family: 'Public Sans'; + font-style: normal; + font-weight: 500; + font-display: swap; + src: url(public-sans-latin-ext-53e02343.woff2) format('woff2'); + unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF; +} +/* latin */ +@font-face { + font-family: 'Public Sans'; + font-style: normal; + font-weight: 500; + font-display: swap; + src: url(public-sans-latin-6e68d19a.woff2) format('woff2'); + unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; +} +/* latin-ext */ +@font-face { + font-family: 'Public Sans'; + font-style: normal; + font-weight: 600; + font-display: swap; + src: url(public-sans-latin-ext-53e02343.woff2) format('woff2'); + unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF; +} +/* latin */ +@font-face { + font-family: 'Public Sans'; + font-style: normal; + font-weight: 600; + font-display: swap; + src: url(public-sans-latin-6e68d19a.woff2) format('woff2'); + unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; +} +/* latin-ext */ +@font-face { + font-family: 'Public Sans'; + font-style: normal; + font-weight: 700; + font-display: swap; + src: url(public-sans-latin-ext-53e02343.woff2) format('woff2'); + unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF; +} +/* latin */ +@font-face { + font-family: 'Public Sans'; + font-style: normal; + font-weight: 700; + font-display: swap; + src: url(public-sans-latin-6e68d19a.woff2) format('woff2'); + unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; +} diff --git a/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-0bc96cd0.woff2 b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-0bc96cd0.woff2 new file mode 100644 index 0000000..090f82f Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-0bc96cd0.woff2 differ diff --git a/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-1b581b15.woff2 b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-1b581b15.woff2 new file mode 100644 index 0000000..67aeeb0 Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-1b581b15.woff2 differ diff --git a/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-e2661bcc.woff2 b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-e2661bcc.woff2 new file mode 100644 index 0000000..0804aaf Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-e2661bcc.woff2 differ diff --git a/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-ext-1969f6ff.woff2 b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-ext-1969f6ff.woff2 new file mode 100644 index 0000000..732c64d Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-ext-1969f6ff.woff2 differ diff --git a/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-ext-93cfae71.woff2 b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-ext-93cfae71.woff2 new file mode 100644 index 0000000..6f33f8c Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-ext-93cfae71.woff2 differ diff --git a/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-ext-a7353317.woff2 b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-ext-a7353317.woff2 new file mode 100644 index 0000000..c0b57de Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/fonts/ibm-plex-mono-latin-ext-a7353317.woff2 differ diff --git a/labs/converged-modeling-patterns/presentations/fonts/public-sans-latin-6e68d19a.woff2 b/labs/converged-modeling-patterns/presentations/fonts/public-sans-latin-6e68d19a.woff2 new file mode 100644 index 0000000..15b5f3d Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/fonts/public-sans-latin-6e68d19a.woff2 differ diff --git a/labs/converged-modeling-patterns/presentations/fonts/public-sans-latin-ext-53e02343.woff2 b/labs/converged-modeling-patterns/presentations/fonts/public-sans-latin-ext-53e02343.woff2 new file mode 100644 index 0000000..49ce111 Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/fonts/public-sans-latin-ext-53e02343.woff2 differ diff --git a/labs/converged-modeling-patterns/presentations/images/image77.png b/labs/converged-modeling-patterns/presentations/images/image77.png new file mode 100644 index 0000000..05bd1b0 Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/images/image77.png differ diff --git a/labs/converged-modeling-patterns/presentations/images/image78.png b/labs/converged-modeling-patterns/presentations/images/image78.png new file mode 100644 index 0000000..b2f3678 Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/images/image78.png differ diff --git a/labs/converged-modeling-patterns/presentations/images/image79.png b/labs/converged-modeling-patterns/presentations/images/image79.png new file mode 100644 index 0000000..8ea5307 Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/images/image79.png differ diff --git a/labs/converged-modeling-patterns/presentations/images/image83.png b/labs/converged-modeling-patterns/presentations/images/image83.png new file mode 100644 index 0000000..1a2d64a Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/images/image83.png differ diff --git a/labs/converged-modeling-patterns/presentations/images/image84.png b/labs/converged-modeling-patterns/presentations/images/image84.png new file mode 100644 index 0000000..a637d2c Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/images/image84.png differ diff --git a/labs/converged-modeling-patterns/presentations/images/image89.png b/labs/converged-modeling-patterns/presentations/images/image89.png new file mode 100644 index 0000000..2786fbd Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/images/image89.png differ diff --git a/labs/converged-modeling-patterns/presentations/images/image93.png b/labs/converged-modeling-patterns/presentations/images/image93.png new file mode 100644 index 0000000..6477ef6 Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/images/image93.png differ diff --git a/labs/converged-modeling-patterns/presentations/images/umt-icon.png b/labs/converged-modeling-patterns/presentations/images/umt-icon.png new file mode 100644 index 0000000..d6e06c2 Binary files /dev/null and b/labs/converged-modeling-patterns/presentations/images/umt-icon.png differ diff --git a/labs/converged-modeling-patterns/run.sh b/labs/converged-modeling-patterns/run.sh new file mode 100755 index 0000000..a67a7e5 --- /dev/null +++ b/labs/converged-modeling-patterns/run.sh @@ -0,0 +1,155 @@ +#!/usr/bin/env bash +# ============================================================================ +# One command: bring up Oracle AI Database 26ai Free (+ ORDS + the Oracle API for +# MongoDB) and validate every pattern across BOTH lanes. +# +# ./run.sh # bring up the DB and run all patterns, both lanes +# ./run.sh 03-bucket # run just one pattern (dir name under patterns/) +# +# SQL lane: each *.sql runs through sqlplus. WHENEVER SQLERROR makes any ORA-/PLS- +# error a non-zero exit -> reported FAIL. +# Mongo lane: each *.js runs through mongosh, in-container. A *parity.js compares the +# SQL-lane result (captured from _capture.sql and injected as $SQL_RESULT) +# against the same data read through the MongoDB API; it exits non-zero on +# mismatch -> reported FAIL. +# +# Idempotent: rerun as often as you like. SQL-only patterns (no *.js) simply skip +# the Mongo lane. +# +# Stage 3: the lab console's vitest suites (app/), in a throwaway container. Skipped +# while cmp-lab-ui runs in event mode (never run this during an event); if a +# (solo) cmp-lab-ui is running it is restarted afterwards so its in-memory +# dirty/built flags and result cache reload from the database. +# ============================================================================ +set -uo pipefail +cd "$(dirname "$0")" + +ORACLE_PASSWORD="${ORACLE_PASSWORD:-Sandbox2026}" +CMP_PASSWORD="${CMP_PASSWORD:-CmpUser2026}" +ADMIN="system/${ORACLE_PASSWORD}@localhost:1521/FREEPDB1" +APP="cmp_user/${CMP_PASSWORD}@localhost:1521/FREEPDB1" +# $external must reach mongosh literally, so it is escaped for the host shell here +# and expanded only inside the container as the value of $MURI. +MURI="mongodb://cmp_user:${CMP_PASSWORD}@localhost:27017/CMP_USER?authMechanism=PLAIN&authSource=\$external&retryWrites=false&loadBalanced=true" +FILTER="${1:-}" + +# Container CLI for exec/inspect: docker, else podman (CONTAINER_ENGINE overrides). +# The stack itself comes up through compose when a provider is available, otherwise +# through ./lab.sh, which runs the same services with the plain CLI (Podman, no compose). +CLI="${CONTAINER_ENGINE:-$(command -v docker >/dev/null && echo docker || echo podman)}" +if docker compose version >/dev/null 2>&1; then COMPOSE="docker compose" +elif podman compose version >/dev/null 2>&1; then COMPOSE="podman compose" +else COMPOSE=""; fi + +dexec() { "$CLI" exec -i cmp-oracle bash -lc "$1"; } # in-container bash +sqlpipe(){ "$CLI" exec -i cmp-oracle bash -lc "sqlplus -s -L $APP"; } # stdin -> sqlplus + +# Record the console's state BEFORE compose touches anything (stage 3 depends on it). +ui_event() { # true when cmp-lab-ui is running with LAB_MODE=event + [ "$("$CLI" inspect -f '{{.State.Running}}' cmp-lab-ui 2>/dev/null)" = true ] && + "$CLI" inspect -f '{{range .Config.Env}}{{println .}}{{end}}' cmp-lab-ui 2>/dev/null | grep -qx 'LAB_MODE=event' +} +ui_event_at_start=0; ui_event && ui_event_at_start=1 + +if [ -n "$COMPOSE" ]; then + echo "==> Bringing up the stack ($COMPOSE up -d) ..." + ORACLE_PASSWORD="$ORACLE_PASSWORD" CMP_PASSWORD="$CMP_PASSWORD" $COMPOSE up -d +else + echo "==> Bringing up the stack (./lab.sh up: no compose provider, plain $CLI) ..." + ORACLE_PASSWORD="$ORACLE_PASSWORD" CMP_PASSWORD="$CMP_PASSWORD" ./lab.sh up || exit 1 +fi + +echo -n "==> Waiting for the database " +until dexec "echo 'select 1 from dual;' | sqlplus -s -L $ADMIN" 2>/dev/null | grep -q '^[[:space:]]*1'; do + echo -n '.'; sleep 5 +done +echo " ready." + +echo "==> Granting schema privileges (idempotent) ..." +dexec "sqlplus -s -L $ADMIN" >/dev/null <<'SQL' +GRANT CREATE SESSION, CREATE TABLE, CREATE VIEW, CREATE PROCEDURE, CREATE TRIGGER, + CREATE SEQUENCE, CREATE MATERIALIZED VIEW, CREATE PROPERTY GRAPH, SODA_APP TO cmp_user; +ALTER USER cmp_user QUOTA UNLIMITED ON users; +EXIT +SQL + +echo -n "==> Waiting for the MongoDB API (ORDS) " +mongo_ok=0 +for i in $(seq 1 60); do + if "$CLI" exec -i -e MURI="$MURI" cmp-oracle bash -lc \ + 'echo "quit(0)" | mongosh "$MURI" --quiet --file /dev/stdin' >/dev/null 2>&1; then + mongo_ok=1; break + fi + echo -n '.'; sleep 5 +done +[ "$mongo_ok" = 1 ] && echo " ready." || echo " NOT ready (Mongo-lane scripts will FAIL and be reported)." + +pass=0; fail=0; failed="" + +run_sql() { # $1 = file + echo "---- [SQL] $1" + { echo "WHENEVER SQLERROR EXIT SQL.SQLCODE"; + echo "SET PAGESIZE 50 LINESIZE 200 FEEDBACK ON LONG 8000"; + cat "$1"; } | sqlpipe + if [ "${PIPESTATUS[1]:-1}" -eq 0 ]; then echo " [PASS] $1"; pass=$((pass+1)); + else echo " [FAIL] $1"; fail=$((fail+1)); failed="$failed $1"; fi +} + +run_js() { # $1 = file, $2 = pattern dir + local js="$1" dir="$2" sqlr="" + echo "---- [MONGO] $js" + if [[ "$js" == *parity.js && -f "$dir/_capture.sql" ]]; then + sqlr=$(cat "$dir/_capture.sql" | sqlpipe 2>/dev/null | sed '/^[[:space:]]*$/d' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + if [ -z "$sqlr" ]; then echo " [FAIL] $js (SQL capture empty)"; fail=$((fail+1)); failed="$failed $js"; return; fi + fi + "$CLI" exec -i -e MURI="$MURI" -e SQL_RESULT="$sqlr" cmp-oracle \ + bash -lc 'mongosh "$MURI" --quiet --file /dev/stdin' < "$js" + if [ "$?" -eq 0 ]; then echo " [PASS] $js"; pass=$((pass+1)); + else echo " [FAIL] $js"; fail=$((fail+1)); failed="$failed $js"; fi +} + +for d in patterns/*/; do + name="$(basename "$d")" + [ -n "$FILTER" ] && [ "$name" != "$FILTER" ] && continue + echo ""; echo "################ $name ################" + # SQL lane first (creates the tables / duality views the Mongo lane reads). + for f in "$d"01-document-model.sql "$d"02-converged.sql; do + [ -f "$f" ] && run_sql "$f" + done + # Mongo lane, in filename order (01 doc-model, 02 sql-in-pipeline, 03 parity). + for js in "$d"*.js; do + [ -f "$js" ] && run_js "$js" "${d%/}" + done +done + +# Stage 3: the hands-on console's own tests (unit + integration) against this database. +# Skipped when a single pattern is requested. +if [ -z "$FILTER" ]; then + echo ""; echo "################ lab console (app/) ################" + if [ "$ui_event_at_start" = 1 ] || ui_event; then + echo " [SKIP] lab-ui tests: cmp-lab-ui is running in EVENT mode. The integration tests" + echo " share its database; run them after the event (see docs/instructor-runbook.md)." + else + # --no-deps: stage 1 already brought the database up; never recreate it from here. + if [ -n "$COMPOSE" ]; then + lab_tests() { $COMPOSE --profile test build lab-ui-test && $COMPOSE --profile test run --rm --no-deps lab-ui-test; } + else + lab_tests() { ORACLE_PASSWORD="$ORACLE_PASSWORD" CMP_PASSWORD="$CMP_PASSWORD" ./lab.sh test; } + fi + if lab_tests; then + echo " [PASS] lab-ui tests"; pass=$((pass+1)) + else + echo " [FAIL] lab-ui tests"; fail=$((fail+1)); failed="$failed lab-ui-tests" + fi + if [ "$("$CLI" inspect -f '{{.State.Running}}' cmp-lab-ui 2>/dev/null)" = true ]; then + echo "==> Restarting lab-ui so it reloads its dirty/built flags and cache ..." + "$CLI" restart cmp-lab-ui >/dev/null + fi + fi +fi + +echo ""; echo "===================================================================" +echo " RESULT: $pass passed, $fail failed (SQL + MongoDB lanes)" +[ -n "$failed" ] && echo " Failed:$failed" +echo "===================================================================" +[ "$fail" -eq 0 ]