feat(core): import wren-engine into core/

.claude/CLAUDE.md

@@ -1,149 +1,113 @@
 # CLAUDE.md
 
-WrenAI is an open-source GenBI (Generative BI) agent that converts natural language questions into SQL queries and charts. It uses a semantic layer (MDL - Metadata Definition Language) to guide LLM-powered text-to-SQL generation via retrieval-augmented generation (RAG).
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
-## Repository Structure
-
-This is a monorepo with three main services:
+## Project Overview
 
-- **wren-ui/** — Next.js 14 frontend + Apollo GraphQL backend (TypeScript, Yarn 4.5.3)
-- **wren-ai-service/** — AI/LLM service (Python 3.12, FastAPI, Poetry, Just command runner)
-- **wren-launcher/** — CLI deployment tool (Go 1.18+, Make)
-- **docker/** — Docker Compose configs for running all services together
-- **deployment/** — Kubernetes/Kustomize manifests
-- **wren-engine/** — SQL engine (git submodule, not developed here)
-- **wren-mdl/** — MDL JSON schema definitions
+Wren is an open-source semantic engine for MCP clients and AI agents. It translates SQL queries through a semantic layer (MDL — Modeling Definition Language) and executes them against 22+ data sources (PostgreSQL, BigQuery, Snowflake, Spark, etc.). The Rust engine is powered by Apache DataFusion (Canner fork).
 
-## Build, Test, and Lint Commands
+The previous WrenAI services (`wren-ui/`, `wren-ai-service/`, `wren-launcher/`, `docker/`, `deployment/`) were moved to the `legacy/v1` branch (tag `v1-final`) as of the wren-engine import. Active development is focused on the Open Context Engine.
 
-### wren-ui (TypeScript/Next.js)
+## Repository Structure
 
-```bash
-cd wren-ui
-yarn install
-yarn dev                # Dev server on port 3000 (TZ=UTC)
-yarn build              # Production build (max-old-space-size=8192)
-yarn lint               # TypeScript type check + ESLint
-yarn check-types        # tsc --noEmit
-yarn test               # Jest unit tests
-yarn test:e2e           # Playwright E2E tests (installs chromium)
-yarn migrate            # Knex database migrations
-yarn rollback           # Knex migration rollback
-yarn generate-gql       # GraphQL codegen from codegen.yaml
+```
+core/
+├── wren-core/        Rust semantic engine (Cargo workspace)
+├── wren-core-base/   Shared Rust crate — manifest types (Model, Column, Metric, Relationship, View) + ManifestBuilder
+├── wren-core-py/     PyO3 bindings exposing wren-core to Python (PyPI: wren-core)
+├── wren-core-wasm/   WebAssembly build of wren-core for in-browser semantic SQL (npm: wren-core-wasm)
+├── wren/             Python SDK and CLI — `wren` command, profile/context/memory management (PyPI: wren-engine)
+└── wren-mdl/         MDL JSON schema definition
+
+docs/core/            Module documentation
+examples/             Example projects (placeholder — to be populated)
+skills/               CLI-based agent skills (wren-generate-mdl, wren-usage, wren-dlt-connector, wren-onboarding)
+scripts/              Repo helper scripts
 ```
 
-Environment: set `DB_TYPE=sqlite` (default) or `DB_TYPE=pg` with PostgreSQL connection vars. The UI needs `WREN_ENGINE_ENDPOINT`, `WREN_AI_ENDPOINT`, and `IBIS_SERVER_ENDPOINT` to connect to backend services.
-
-### wren-ai-service (Python/FastAPI)
+## Build & Development Commands
 
+### core/wren-core (Rust)
 ```bash
-cd wren-ai-service
-poetry install
-just init               # Creates config.yaml and .env.dev from examples
-just up                 # Start dev Docker services (Qdrant, engine, etc.)
-just start              # Run AI service (poetry run python -m src.__main__)
-just test               # pytest (spins up Docker deps, ignores usecases)
-just test [test_args]   # e.g., just test tests/pytest/pipelines/
-just test-usecases      # Run use-case integration tests
-just down               # Stop Docker services
-just load-test          # Locust load tests
+cd core/wren-core
+cargo check --all-targets                                  # compile check
+cargo test --lib --tests --bins                            # tests (set RUST_MIN_STACK=8388608)
+cargo fmt --all                                            # format
+cargo clippy --all-targets --all-features -- -D warnings   # lint
+taplo fmt                                                  # format Cargo.toml files
 ```
 
-Configuration is via `config.yaml` (multi-document YAML with sections for LLM, embedder, engine, document_store, pipeline, and settings). Environment variables in `.env.dev` (API keys). Settings load order: defaults → env vars → .env.dev → config.yaml.
+Most unit tests live in `core/wren-core/core/src/mdl/mod.rs`. SQL end-to-end tests use sqllogictest files in `core/wren-core/sqllogictest/test_files/`.
 
-Pre-commit hooks: `poetry run pre-commit install` then `poetry run pre-commit run --all-files`
+### core/wren-core-py (Python bindings)
+```bash
+cd core/wren-core-py
+just install      # poetry install
+just develop      # build dev wheel with maturin
+just test-rs      # Rust tests (cargo test --no-default-features)
+just test-py      # Python tests (pytest)
+just test         # both
+just format       # cargo fmt + ruff + taplo
+```
 
-### wren-launcher (Go)
+### core/wren-core-wasm (WASM)
+```bash
+cd core/wren-core-wasm
+just build        # wasm-pack build (browser target)
+just test         # wasm-pack test
+```
+Outputs a ~68 MB WASM binary; distributed via npm and unpkg (jsDelivr's 50 MB per-file CDN limit blocks it).
 
+### core/wren (SDK & CLI)
 ```bash
-cd wren-launcher
-make build              # Cross-compile for macOS/Linux/Windows
-make test               # go test ./...
-make check              # fmt + vet + lint (golangci-lint)
-make lint-fix           # Auto-fix lint issues
+cd core/wren
+just install              # build wren-core-py wheel + uv sync
+just install-all          # with all optional extras (incl. memory)
+just install-extra <e>    # e.g. just install-extra postgres
+just install-memory       # memory extra (lancedb + sentence-transformers)
+just dev                  # run `wren` CLI
+just test                 # pytest tests/
+just test-memory          # memory-specific tests
+just lint                 # ruff format --check + ruff check
+just format               # ruff auto-fix
+just build                # uv build (produces wheel)
 ```
 
-## Architecture
+Uses `uv` (not Poetry). `pyproject.toml` uses `hatchling` as build backend. Optional extras: `postgres`, `mysql`, `bigquery`, `snowflake`, `clickhouse`, `trino`, `mssql`, `databricks`, `redshift`, `spark`, `athena`, `oracle`, `memory`, `all`, `dev`.
 
-### Service Communication Flow
+## Architecture: Query Flow
 
 ```
-User → Wren UI (Next.js :3000)
-         ↓ GraphQL (Apollo Server embedded in Next.js API routes)
-       Apollo Server → Wren AI Service (FastAPI :5556) [HTTP REST]
-                     → Wren Engine (:8080) [SQL validation/execution]
-                     → Ibis Server (:8000) [SQL abstraction for data sources]
-       Wren AI Service → Qdrant (:6333) [vector search for RAG]
-                       → LLM Provider (OpenAI/Azure/etc.) [text-to-SQL generation]
+SQL query
+  → wren CLI / wren-core-py
+  → wren-core (Rust): MDL analysis → logical plan → optimization
+  → DataFusion (query planning, Canner fork canner/v49.0.1)
+  → connector-specific SQL (Ibis / sqlglot)
+  → native execution on the target data source
 ```
 
-### Wren UI Internal Architecture
-
-The Next.js app embeds an Apollo GraphQL server in its API routes (`src/apollo/`):
-
-- **`src/apollo/server/resolvers/`** — GraphQL resolvers (asking, model, project, dashboard, etc.)
-- **`src/apollo/server/services/`** — Business logic layer (askingService, deployService, mdlService, queryService, etc.)
-- **`src/apollo/server/repositories/`** — Data access layer using Knex (SQLite or PostgreSQL)
-- **`src/apollo/server/adaptors/`** — External service adapters (AI service, engine)
-- **`src/apollo/client/`** — Frontend GraphQL operations
-- **`src/components/`** — React components organized by page (home, setup, modeling, knowledge)
-- **`src/pages/`** — Next.js page routes
+## Key Architecture Details
 
-Path aliases: `@/*` → `./src/*`, `@server/*` → `./src/apollo/server/*`
-
-### Wren AI Service Internal Architecture
-
-The Python service uses a pipeline-based architecture:
-
-- **`src/pipelines/`** — RAG pipeline implementations:
-  - `indexing/` — MDL schema, table descriptions, historical questions, SQL pairs → Qdrant
-  - `retrieval/` — Semantic search for relevant context from Qdrant
-  - `generation/` — SQL generation, chart generation, intent classification
-  - `ask/` — Orchestrates retrieval + generation for text-to-SQL
-  - `ask_details/` — SQL breakdown and explanation
-  - `semantics/` — Semantic processing helpers
-- **`src/web/v1/services/`** — Service layer (AskService, SemanticsPreparationService, ChartService, SqlPairsService, etc.)
-- **`src/web/v1/routers/`** — FastAPI route handlers
-- **`src/core/`** — Base abstractions (pipeline, provider, engine interfaces)
-- **`src/globals.py`** — ServiceContainer wiring all services and pipelines together
-- **`src/config.py`** — Pydantic Settings with all configuration knobs
-
-Pipelines are configured declaratively in `config.yaml`, wiring LLM providers, embedders, document stores, and engines to named pipeline components.
-
-### Data Flow for "Ask" (Text-to-SQL)
-
-1. User submits natural language question in UI
-2. UI sends GraphQL mutation to Apollo Server
-3. Apollo Server calls AI Service REST API
-4. AI Service runs intent classification → retrieves relevant schema/context from Qdrant → generates SQL via LLM
-5. Generated SQL is validated against Wren Engine
-6. SQL corrections are attempted if validation fails (up to `max_sql_correction_retries`)
-7. Results returned through the chain back to UI
-
-### MDL (Metadata Definition Language)
-
-The semantic layer that maps business concepts to database schema. Defines models, columns, relationships, metrics, and calculated fields. MDL is indexed into Qdrant as vector embeddings to provide context for LLM SQL generation. Schema defined in `wren-mdl/mdl.schema.json`.
-
-## Docker Development
-
-To run the full stack locally:
-```bash
-cd docker
-cp .env.example .env.local    # Configure API keys and ports
-cp config.example.yaml config.yaml
-docker compose --env-file .env.local up -d
-```
+**wren-core internals** (`core/wren-core/core/src/`):
+- `mdl/` — Core MDL processing: `WrenMDL` (manifest + symbol table), `AnalyzedWrenMDL` (with lineage), function definitions per dialect (scalar/aggregate/window), type planning
+- `logical_plan/analyze/` — DataFusion analyzer rules: `ModelAnalyzeRule` (TableScan → ModelPlanNode), scope tracking, access control (RLAC/CLAC), view expansion, relationship chain resolution
+- `logical_plan/optimize/` — Optimization passes: type coercion, timestamp simplification
+- `sql/` — SQL parsing and analysis
 
-For developing a single service while others run in Docker, use `docker-compose-dev.yaml` in the AI service's `tools/dev/` directory.
+**Manifest types** (`core/wren-core-base/src/mdl/`):
+- `manifest.rs` — `Manifest`, `Model`, `Column`, `Metric`, `Relationship`, `View`, `RowLevelAccessControl`, `ColumnLevelAccessControl`
+- `builder.rs` — Fluent `ManifestBuilder` API
+- Uses `wren-manifest-macro` for auto-generating Pydantic-compatible Python classes
 
-## Commit Convention
+## Known wren-core Limitations
 
-Follows conventional commits: `type(scope): description`
-- Scopes: `wren-ui`, `wren-ai-service`, `wren-launcher`
-- Types: `feat`, `fix`, `chore`, `refactor`
-- Examples: `feat(wren-ui): add dashboard widget`, `fix(wren-ai-service): handle empty MDL`
+**ModelAnalyzeRule — correlated subquery column resolution**: cannot resolve outer column references inside correlated subqueries; only sees the subquery's own table scope. Affects TPCH Q2, Q4, Q15, Q17, Q20, Q21, Q22.
 
-## CI/CD
+## Conventions
 
-- PR labels trigger service-specific CI: `ci/ui` for UI tests/lint, `ci/ai-service` for AI service tests
-- Docker images published to `ghcr.io/canner/`
+- **Commits**: Conventional commits (`feat:`, `fix:`, `chore:`, `refactor:`, `test:`, `docs:`, `perf:`, `deps:`). Releases are automated via release-please with independent release lines per module.
+- **Rust**: format with `cargo fmt`, lint with `clippy -D warnings`, TOML with `taplo`.
+- **Python**: format and lint with `ruff` (line-length 88, target Python 3.11). `core/wren-core-py` uses Poetry; `core/wren` uses uv.
+- **DataFusion fork**: `https://github.com/Canner/datafusion.git` branch `canner/v49.0.1`.
+- **Snapshot testing**: wren-core uses `insta` for Rust snapshot tests.
+- **CI**: Per-module path-filtered workflows trigger only on changes inside that module.

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "wren-engine"]
-	path = wren-engine
-	url = git@github.com:Canner/wren-engine.git

CHANGELOG.md

@@ -0,0 +1,11 @@
+# Changelog
+
+This file tracks repository-level changes (layout, licensing, governance). Per-package release notes live in each package's own `CHANGELOG.md` (e.g. `core/wren/CHANGELOG.md`, `core/wren-core-py/CHANGELOG.md`) and are managed by release-please.
+
+## 2026-Q2 — Repo consolidation
+
+- **Imported** [`Canner/wren-engine`](https://github.com/Canner/wren-engine) into [`core/`](./core) via `git filter-repo` rewrite, preserving authorship and PR history (~5,300 commits across `core/wren-core`, `core/wren-core-base`, `core/wren-core-py`, `core/wren-core-wasm`, `core/wren`).
+- **Relocated** the existing `wren-mdl/` schema to `core/wren-mdl/`.
+- **Archived** the legacy WrenAI GenBI app (`wren-ai-service`, `wren-ui`, `wren-launcher`, `docker`, `deployment`) on the [`legacy/v1`](https://github.com/Canner/WrenAI/tree/legacy/v1) branch (tag `v1-final`).
+- **Adopted** a multi-license layout (Apache 2.0 for code/skills/SDKs/examples; CC BY 4.0 for docs; AGPL 3.0 reserved for future modules). See [LICENSE](./LICENSE).
+- **Added** root metadata: `README.md`, `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, `SECURITY.md`, this `CHANGELOG.md`.