Skip to content

architecture.md

Latest commit

 

History

History
112 lines (83 loc) · 10.5 KB

architecture.md

File metadata and controls

112 lines (83 loc) · 10.5 KB
Raw

Architecture

Authoritative detail lives in SPEC.md §2–§3. This page is the map.

Crates and dependency direction

Arrows point from a crate to what it depends on. The graph must stay acyclic. Grey nodes are external.

flowchart BT
  core[oxidelake-core<br/>EngineError · BackendKind · BufferLayout · BatchStream · TelemetryHub]
  memory[oxidelake-memory<br/>AlignedBuf · pinned/UMA allocators · SpillManager] --> core
  device[oxidelake-device<br/>GpuBackend · Cpu/Cuda/Metal · HardwareDetector] --> memory
  compute[oxidelake-compute<br/>Gpu*Exec · CPU reference · conformance] --> device
  storage[oxidelake-storage<br/>Parquet config · Arrow IPC spill · io_uring ObjectStore] --> memory
  planner[oxidelake-planner<br/>HardwarePlacementRule · OxidePhysicalCodec] --> compute
  tui[oxidelake-tui<br/>dashboard · oxidelake-tui-demo] --> core
  runtime[oxidelake-runtime<br/>OxideSession · scheduler/worker wrappers · oxide CLI] --> planner
  runtime --> storage
  runtime --> tui
  runtime --> ballista[Apache DataFusion Ballista 54<br/>scheduler · executors · Flight shuffle]
  api[oxidelake-api<br/>DataFrame · SQL · optional PyO3] --> runtime
  classDef ext fill:#e5e7eb,stroke:#9ca3af,color:#111827;
  class ballista ext
Loading

Execution flow

flowchart TB
  q[SQL / DataFrame] --> lp[DataFusion logical plan]
  lp --> pp[Physical plan]
  pp --> hpr[HardwarePlacementRule<br/>rewrite eligible nodes to Gpu*Exec]
  hpr -->|embedded mode| local[In-process pipeline<br/>Arc&lt;RecordBatch&gt; over bounded mpsc]
  hpr -->|cluster mode| sched[Ballista scheduler<br/>our session builder installs the rule<br/>our codec serializes Gpu*Exec]
  sched --> e1[Ballista executor<br/>Gpu*Exec picks the local backend]
  sched --> e2[Ballista executor<br/>Gpu*Exec picks the local backend]
  e1 <-->|Arrow Flight shuffle - Ballista| e2
  e1 --> result[Result stream]
  e2 --> result
Loading

Placement is a physical optimizer rule, not an AST pass: eligibility depends on physical operator shape and resolved types (ADR-0010). Anything outside the bounded v1 GPU coverage stays on CPU. In cluster mode the rule runs on the scheduler against the declared cluster capability (OXIDE_CLUSTER_BACKEND); each Gpu*Exec then selects the real local backend at execute time, so a heterogeneous cluster stays correct (ADR-0013).

Bounded v1 GPU coverage

Operator GPU-eligible shape Otherwise
filter + project =, <, <=, >, >= combined with AND over Int64/Float64 vs literals; column selection; stream-compacted output CPU
hash join inner join, single Int64 key; open-addressing table built with atomic CAS, probed in parallel CPU
aggregate SUM/COUNT/MIN/MAX over Int64/Float64, one low-cardinality Int64 key CPU
vector distance L2 and cosine, query vector vs FixedSizeList<Float32> column CPU

CUDA implements all four; Metal implements filter + project and vector distance in v1.

Memory model and spill tiers

flowchart LR
  vram[Device VRAM<br/>128 B aligned] <-->|demote / promote| host[Pinned or pageable host RAM<br/>64 B aligned]
  host <-->|Arrow IPC spill files via ObjectStore| disk[Local spill directory<br/>LocalFileSystem or UringLocalFileSystem]
Loading
  • Every in-memory table is an Arrow RecordBatch; host buffers are 64-byte aligned, device-bound buffers 128-byte aligned.
  • Pinned host memory is available: with cuda active and a driver present, GpuBackend::alloc_pinned_host allocates page-locked memory (the cuMemHostAlloc path) for DMA without staging copies, falling back to AlignedBuf otherwise. The path taken is observable as MemoryClass::{Pinned, Pageable}.
  • The operator layer does not use it in 0.x. upload_bytes copies from a pageable Arrow Buffer and download_column copies into a pageable Vec, so the operator path pays the staging copy that pinned memory exists to avoid; alloc_pinned_host is exercised only by the T4 device tests. Pinned staging for operator transfers is audit item P6 (#26), tracked in the roadmap's Phase 8. Any transfer-throughput number measured today is a pageable number.
  • On Apple Silicon, MTLResourceStorageModeShared buffers give CPU and GPU one physical allocation.
  • SpillManager holds per-tier budgets, demotes cold batches on high-watermark pressure, promotes on access, and exports metrics to the TUI. On a GPU-less machine the device tier is empty and the host ↔ disk path is fully testable.
  • It is a library, not part of the query path (#25). No operator registers a batch with it, so in a running query the tier gauges and the spill counters stay at zero however much memory the query used — a reading of zero there says nothing about the query. The dashboard states this rather than implying otherwise: the Telemetry panel's last line reads spill library only: no query registers batches, and TelemetrySnapshot::capacity.spill_on_query_path is the flag it renders. The gauges' capacities now come from the local backend's MemoryInfo (the device's total for a GPU backend, host RAM for the CPU one) instead of the fixed constants they used to be drawn against, and a tier the engine has no number for is drawn without a ratio.
  • Wiring it in is not a matter of calling it from the join. GpuHashJoinExec's build side is probed by every batch, so demoting it would trade an unbounded memory claim for an unbounded time one, and promoting it once puts it straight back in host RAM; GpuAggregateExec collects its whole input before aggregating, so bounding it needs the streaming aggregate first. The honest version of this in 0.2 is the label; the engineering is Phase 8.

Hardware abstraction

Arc<dyn GpuBackend> is selected at startup by HardwareDetector (CUDA → Metal → CPU, overridable with OXIDE_BACKEND). The trait is object-safe and synchronous; asynchrony lives on device streams (ADR-0005). CpuBackend is always present and fully functional — it is the semantic reference the GPU backends must match in the conformance suite.

Kernels ship as source: .cu compiled at runtime with NVRTC (PTX cached per device), .metal compiled with newLibraryWithSource. No nvcc, no Xcode step (ADR-0003, ADR-0004).

Storage

There is no private file format (ADR-0014).

flowchart LR
  pq[Parquet lake data<br/>statistics · page index · Bloom filters] --> df[DataFusion Parquet scan<br/>prunes row groups and pages]
  df --> os[object_store::ObjectStore<br/>LocalFileSystem | UringLocalFileSystem]
  spill[SpillManager] <-->|Arrow IPC, 64 B aligned, no decode| os
  os --> nvme[(Local NVMe)]
Loading
  • Lake data is Parquet through DataFusion's listing tables; oxidelake-storage owns the writer properties (row-group size, dictionaries, Bloom filters on keys, page statistics), the session pruning settings, and the tests that prove pruning happened via scan metrics while results stay identical.
  • Spill files and the optional hot cache are Arrow IPC: raw Arrow buffers, zero decode.
  • The io_uring fast path is an ObjectStore implementation behind the io-uring feature; both stores pass one conformance test (ADR-0007).

Telemetry and TUI

TelemetryHub (in oxidelake-core) is the only coupling between engine and dashboard; in v1 it observes the local process only. The TUI is a state machine with a pure render(state, frame); repaints are bracketed in DEC 2026 synchronized updates. Four panels: Plan DAG with hardware tags, Inspector, Telemetry gauges, Describe (P25/P50/P99).

Observability

Three surfaces, in increasing cost to the reader (#33):

  • One INFO line per query, from OxideSession::collect: query finished mode=embedded/cuda rows=98 batches=1 elapsed_ms=3.9 gpu_operators=2 fallback_batches=2. It is on collect and not on sql, because a DataFrame has not run yet. Logs go to stderr, so they never mix into oxide sql --output json, and ANSI is off unless stderr is a terminal.
  • One span per (operator, partition), oxide.operator, entered around each batch, with fields operator, partition, target (what the plan says) and backend (what is executing). The pair is the question these logs exist to answer; they differ exactly when the CPU-fallback counter is moving. GpuOperator's own methods run inside it and add no span of their own — a nested span there would carry no field the outer one does not.
  • A Prometheus endpoint, oxide-worker --metrics-port / oxide-scheduler --metrics-port (feature metrics): oxide_operator_* counters labelled by operator and backend, plus the tier gauges and spill counters. A cluster executor never runs the placement rule, so its operators are attached to a process-wide TelemetryHub by the plan codec; that hub is what is served. It is unauthenticated, like the Ballista ports themselves — bind it on a private interface. Without the feature the flag is refused rather than ignored.

OxideSession::telemetry() is the embedded hub and stays empty for a cluster session: the work happened in other processes, and each of those has its own endpoint.

The Inspector's cpu fallback line is the one counter that cannot be inferred from the others (#32). A GPU deployment that runs every batch on the CPU reference produces identical rows and identical EXPLAIN tags, so fallback_batches / batches is what tells the two apart; it is green at zero, yellow when some batches fell back and red when all of them did. The matching planner-side answer is the placement notes section oxide explain prints under the plan, naming each node the rule left on the CPU and why — a skipped node is an ordinary DataFusion operator in the plan above it, indistinguishable from one that was never eligible. Tested two ways: in-process with ratatui::backend::TestBackend + insta, and end-to-end with termlens driving the deterministic oxidelake-tui-demo binary in a real PTY (ADR-0011).

Out of scope for v1

GPUDirect Storage / cuFile and GPU-side Parquet decode · tuning Ballista's retry or AQE behaviour · TLS/auth for the Ballista cluster · catalog/metastore (Iceberg, Delta) · transactions · cost-based optimization beyond DataFusion defaults · Lance integration (v2) · cluster-wide telemetry in the TUI · Windows.