Start here. Everything in this folder is written so that someone who has never seen the code can check the claims against the source, the tests and the recorded benchmark runs: each page names the file, the test or the CSV a claim comes from.
| Read this | To learn |
|---|---|
| ../README.md | What ArrowMetal is, the pitch, headline numbers, quick start in Swift and Python |
| ARROW_FUNCTIONS.md | Coverage of Apache Arrow compute by exact name: all 307 v25 function names and the 5 added in v26, one row each, generated from a registry the test suite executes against pyarrow.compute |
| COVERAGE.md | The same ground by family, with the Arrow type matrix and interop: which functions and types are supported, and how (GPU, GPU/CPU, CPU, partial) |
| EVALUATION.md | How correctness is checked: CPU oracles, the scenario matrix, bit-exact IEEE-754 tests, and the differential harness against pyarrow.compute, with open findings |
| UPSTREAM.md | What the matrix and the bindings found in other projects (pyarrow, the Arrow tz database, Apple Metal, the Arrow JS, R and Go libraries, the Swift compiler), the test behind each finding, and where every upstream report stands |
| TESTING.md | Every test suite, what it covers, and the commands to run it |
| BENCHMARKS.md | Every benchmark round with hardware, methodology, wall time and CPU time; the tables the README quotes |
| TO_IMPROVE.md | Every row of the matrix short of the 3x bar, the rows where the CPU idiom is ahead first, grouped by the measured cause (ten sections: dispatch floor, memory-bound single passes, software binary64, host regex, CPU views, temporal extraction, unique and value_counts, grouped moments, variable-length string output, whole-query chains), plus the rows under the 3x bar and what changed since the previous matrix |
| CROSSOVER.md | The crossover: from what row count the GPU path is ahead of the fastest CPU idiom, per operation and per family, and ahead of a single-core CPU loop (the bench's, from which the router's table was fitted); generated by Benchmarks/crossover.py from a size sweep of the matrix and arrowmetal-bench crossover |
| APPLE_REPORTS.md | The three Metal findings reported to Apple through Feedback Assistant on 2026-09-20 (64-bit atomics against the Feature Set Tables, pipeline creation on GitHub's virtual GPU, CPU-to-GPU visibility for a persistent kernel): what each says, what was measured, what was asked, and the FB numbers; no attachments, no personal data |
| BENCHMARKS_MATRIX.md | The complete operation-by-operation comparison against Polars, pyarrow, pandas and numpy in both their eager and most-parallel idioms, at 1k to 50M rows, with a verdict per row and a list of everything below the 3x bar |
| DESIGN.md | How it works: buffers, kernels, batching, GPU-side lengths, software Float64, the CPU/GPU router (which small inputs run on a byte-identical CPU loop, and how to override it), where the time goes |
| PANDAS.md | pandas on the GPU: the .am accessor and the zero-code-change accel mode, what routes to the GPU and when, the zero-copy and null rules, the numbers and the limits |
| NUMPY.md | numpy on the GPU through pyarrow: which dtypes cross without a copy (asserted by test_numpy.py), NaN-as-value semantics, the matrix rows measured against numpy, and what numpy dispatch is not implemented |
| SWIFT.md | The native Swift API: install, the types, the copy rule, tests and where its numbers live |
| C.md | The C ABI: build the dylib, the handle and error conventions, an example, what the header covers |
| RUST.md | The Rust crate over the C ABI: install, the copy rule as measured, what is wrapped and what is not, the timing table with its method |
| GO.md | The Go module: cgo shim, pinning, the alignment finding, what is wrapped, the timing table |
| TYPESCRIPT.md | The Node addon and TypeScript API: Arrow JS interop, lifetimes, the copy rule, timings |
| R.md | The R package: arrow R interop, int64 through bit64, the timing table with both measurement modes |
| EXPR.md | Fused expression queries: one runtime-generated kernel for a whole expression DAG, the grammar, how nulls are compiled, the numbers and the limits |
| ENGINE.md | The lazy query engine: the logical plan, the optimizer rules with explain() examples, fusion planning, the full join matrix (multi-key, utf8, outer, semi/anti, as-of), window functions, the plan grammar, the numbers and the limits |
| POLARS.md | Polars on the GPU in four tiers: the zero-copy bridge and .arrowmetal namespaces, the Rust expression plugin for lazy plans, the streaming hand-off, and MetalEngine, which runs parts of the optimised plan itself through lf.collect(engine=am.MetalEngine()) — with install, numbers, what the engine translates, where it matches Polars by construction, and how its defaults were measured |
| DUCKDB.md | Using ArrowMetal from DuckDB: the Python bridge (copy-free where DuckDB returns one chunk), the loadable SQL extension, the optimizer extension that runs eligible aggregates of unchanged SQL on the GPU (§4b), streaming tables larger than memory, and which shapes the GPU is ahead on and which it is not |
| DATAFUSION.md | Apache DataFusion 55.1 with a physical optimizer rule that runs full ORDER BY sorts of unchanged SQL on the GPU: registering it, what the default takes and leaves, the semantics matched, the differential grid, rule off against rule on from 250,000 to 50M rows, and the shapes where the rule is behind DataFusion |
| LAKEHOUSE.md | Delta Lake and Apache Iceberg tables: log and metadata replay on the CPU, pruning by partitions and statistics, data files through the GPU Parquet reader; what is supported, what is refused by name, and where results differ from deltalake and pyiceberg |
| STREAMING.md | Out-of-core streaming execution: datasets larger than memory flowing from disk through the GPU, the three-stage pipeline and its measured overlap, every streaming operator with exact-or-approximate marked, and the buffer budget |
| DECISIONS.md | Why it is built this way, one dated entry per decision |
| FINDINGS.md | Things learned the hard way: toolchain quirks, Metal limits, bugs and their lessons |
| RESIDENT.md | The resident-kernel experiment: where the ~65 µs per command buffer goes, and the two measured probes showing a persistent GPU worker polling shared memory does not beat it |
| ROADMAP.md | The project's scope, and where its measured record is kept |
| ../CHANGELOG.md | What each release delivered |
| ../Benchmarks/README.md | The benchmark programs and the fairness rules |
| ../python/README.md | The Python package: install, wheel build, usage |
| RELEASE.md | How a release is built and verified |
| ../CONTRIBUTING.md | How to add a kernel, test it, and benchmark it |
Both install routes are below.
# Swift: add the package by path or git URL in Package.swift, then
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift build -c release
# Python from source
swift build -c release --product ArrowMetalC
PYTHONPATH=python python -c "import arrowmetal as am; print(am.device_name())"
# Python from a wheel that bundles the dylib (macOS 14+, arm64)
pip install build && scripts/build_wheel.sh # or python/build_wheel.sh, if the dylib is built
pip install python/dist/arrowmetal-0.5.0-*.whl
pip install 'python/dist/arrowmetal-0.5.0-*.whl[polars,duckdb,pandas]' # optional bridges
The package finds libArrowMetalC.dylib in one of three places, in order: $ARROWMETAL_LIB, which pins
one specific build and wins over everything,
then the copy bundled inside the wheel (arrowmetal/_lib/), then a development build in .build/release
beside a source checkout. See ../python/README.md.
All of it runs on an Apple silicon Mac on macOS 14 or later (the package's floor) with Xcode installed:
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test # Swift suite, debug
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test -c release # same, optimised
swift build -c release --product ArrowMetalC && PYTHONPATH=python python -m pytest python/tests -q
swift run -c release arrowmetal-bench # Metal vs all CPU cores vs Accelerate, with CPU-ms
python Benchmarks/python_bench.py # Polars / pyarrow / pandas on the same data shapes
PYTHONPATH=python python Benchmarks/python_gpu_bench.py # ArrowMetal from Python vs Polars, in-process
PYTHONPATH=python python Benchmarks/expr_bench.py # fused expression queries vs one kernel per operator
PYTHONPATH=python python Benchmarks/engine_bench.py # the lazy query engine vs Polars lazy and DuckDB
cd polars-plugin && cargo build --release # the Polars expression plugin (tier 2)
PYTHONPATH=python python -m pytest python/tests/test_polars.py -q # the three Polars tiers
PYTHONPATH=python python Benchmarks/polars_bench.py # native Polars vs both Polars tiers
PYTHONPATH=python python Benchmarks/pandas_bench.py # plain pandas vs the .am accessor vs accel mode
PYTHONPATH=python python Benchmarks/streaming_bench.py --data-dir /tmp/am-stream --size-gb 30
# out-of-core streaming vs Polars, DuckDB and pyarrow.dataset
swift run -c release arrowmetal-examples # six end-to-end scenarios
Every benchmark table in the repository names the machine it was measured on. Numbers from GitHub-hosted runners are never published: their GPU is virtual.
| Read this | To learn |
|---|---|
| PARQUET.md | The Parquet reader that decodes on the Apple GPU: the pipeline, the parallel RLE strategy, GPU Snappy and LZ4, the supported encoding/codec/type matrix, structs, maps and lists nested to any depth, the stored ARROW:schema, projection, statistics pushdown with page-index and bloom-filter skipping, benchmarks against pyarrow / Polars / pandas (and a nested-read script against pyarrow / Polars / DuckDB), the small writer, and the limits |
| CSV.md | The CSV reader that parses on the Apple GPU: the quote-aware structure scan, pyarrow's inference rules and how they were established, the GPU float parse and its bit-identity tests, the options, the exact differences from pyarrow.csv.read_csv, and the limits |
| JSON.md | The newline-delimited JSON reader that parses on the Apple GPU: the structure scan, the per-record walk, key matching, type inference, pyarrow's behaviour as probed, the documented differences, the benchmark script and the limits |
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test --filter "ParquetTests|ParquetWriterTests"
PYTHONPATH=python python -m pytest python/tests/test_parquet.py -q # every fixture vs pyarrow.parquet
PYTHONPATH=python python Benchmarks/parquet_bench.py --rows 50000000 # vs pyarrow, Polars, pandas
python Tests/Fixtures/generate_parquet.py # regenerate the committed fixtures