Prepare v0.8.0: larger-than-memory Parquet (pushdown, write options, …

.github/workflows/ci.yml

@@ -5,6 +5,7 @@ on:
     branches: [main]
   pull_request:
     branches: [main]
+  workflow_dispatch:
 
 # Skip NIF for all jobs by default — no download, no compile, just stubs.
 # The test-rust job overrides this with EX_ARROW_BUILD=1 and EX_ARROW_SKIP_NIF=false.
@@ -56,12 +57,8 @@ jobs:
         env:
           MIX_ENV: dev
 
-      - name: Check Livebooks parse
-        run: |
-          for f in livebook/*.livemd; do
-            echo "Checking $f"
-            grep -q '```elixir' "$f" || true
-          done
+      - name: Check Livebook Hex pins and fences
+        run: bash script/check_livebooks.sh
 
   # ── Dialyzer ─────────────────────────────────────────────────────────────────
   dialyzer:
@@ -177,8 +174,56 @@ jobs:
         if: ${{ !matrix.coverage }}
         run: mix test
 
+      - name: Smoke livebook/00_quickstart.exs
+        if: ${{ !matrix.coverage }}
+        run: elixir livebook/00_quickstart.exs
+
       - name: Run coverage
         if: ${{ matrix.coverage }}
         run: mix coveralls.github --include adbc_package:true
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+  # ── Mix.install optional-dep smoke (compile only, no NIF) ───────────────────
+  mix-install:
+    name: Mix.install optional-dep smoke
+    runs-on: ubuntu-latest
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      EX_ARROW_SKIP_NIF: "1"
+    steps:
+      - uses: actions/checkout@v6.0.3
+      - name: Set up Elixir
+        uses: erlef/setup-beam@v1.24.0
+        with:
+          elixir-version: "1.20"
+          otp-version: "29"
+      - name: Smoke Mix.install combinations
+        run: bash script/mix_install_smoke.sh
+
+  # ── Optional apache/arrow-testing corpus (manual / non-blocking) ────────────
+  # Local-first suite; also runnable here via workflow_dispatch. Does not gate PRs.
+  arrow-testing:
+    name: arrow-testing (optional)
+    if: github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    env:
+      EX_ARROW_SKIP_NIF: "false"
+      EX_ARROW_BUILD: "1"
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - uses: actions/checkout@v6.0.3
+      - name: Set up Elixir
+        uses: erlef/setup-beam@v1.24.0
+        with:
+          elixir-version: "1.20"
+          otp-version: "29"
+      - name: Install Rust
+        uses: dtolnay/rust-toolchain@stable
+      - run: mix deps.get
+      - run: mix compile --warnings-as-errors
+      - name: Fetch apache/arrow-testing fixtures
+        run: bash script/fetch_arrow_testing.sh
+      - name: Run arrow_testing suite
+        run: mix test --include arrow_testing

.gitignore

@@ -56,3 +56,5 @@ checksum-Elixir.ExArrow.Native.exs
 
 .DS_Store
 /baoulo/
+
+.tool-versions

CHANGELOG.md

@@ -5,6 +5,39 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.8.0] - 2026-08-21
+
+### Added
+
+- **Parquet read pushdown**: `ExArrow.Stream.from_parquet/2` and
+  `Parquet.Reader.from_file/2` / `from_binary/2` accept `:columns`,
+  `:row_groups`, and `:filters` (predicate AST with statistics pruning).
+  `Parquet.Reader.read_stats/1` reports selected vs skipped row groups.
+- **Parquet write options**: `:compression` (`:snappy`, `:zstd`,
+  `{:zstd, level}`, `:lz4`, `:gzip`, `:none`, alias `:uncompressed`),
+  `:row_group_size`, and `:dictionary` on `Writer.to_file/4` and
+  `to_binary/3`.
+- **`ExArrow.Parquet.Metadata`**: footer-only metadata via `from_file/1` and
+  `from_binary/1` (row groups, column stats, key-value metadata).
+- **Multi-file Parquet streams**: `Stream.from_parquet_files/2` and
+  `from_parquet_dir/2` — lazy per-file open with schema checking;
+  `Stream.close/1` releases the backing Agent when abandoning early.
+- **`ExArrow.IPC.File.write/3`**: public IPC file-format writer (wraps existing NIF).
+- **`ExArrow.RecordBatch.concat/1`**: public batch concatenation (wraps existing NIF).
+- **CI Mix.install optional-dep matrix** and a Parquet Livebook with
+  PyArrow-equivalent examples. Optional apache/arrow-testing fixtures are
+  local-only by default (`script/fetch_arrow_testing.sh` +
+  `mix test --include arrow_testing`); an optional `workflow_dispatch`
+  CI job can run the same suite without gating PRs.
+
+### Changed
+
+- Enabled Parquet compression codecs (`snap`, `zstd`, `lz4`, `gzip`) in the
+  native crate so write options work end-to-end. Includes a DuckDB-generated
+  ZSTD fixture and regression test for third-party compressed reads
+  (thanks [@mindreframer](https://github.com/mindreframer) —
+  [#243](https://github.com/thanos/ex_arrow/pull/243)).
+
 ## [0.7.2] - 2026-07-08
 
 ### Fixed

README.md

@@ -9,7 +9,15 @@
 
 Native Apache Arrow for the BEAM: IPC streaming, Arrow Flight, Arrow Flight SQL, ADBC database bindings, and Arrow-native pipelines. Column data lives in Rust buffers; Elixir holds lightweight opaque handles. Precompiled NIFs for Linux, macOS, and Windows — no Rust required to use.
 
-> **v0.7.2 — stability release.** Fixes optional `:gen_stage` compile issues in some `Mix.install` / cached-build paths. For feature overview, see [What's changed in v0.7.0](#whats-changed-in-v070).
+> **v0.8.0 — Larger-than-memory Parquet.** Column/predicate/row-group pushdown,
+> write compression options, footer metadata, and multi-file directory streams.
+> See [Parquet: read and write](#parquet-read-and-write) and the
+> [Parquet guide](https://ex-arrow.hexdocs.pm/parquet_guide.html).
+>
+> **Related package:** the pure-Elixir [`arrow`](https://hex.pm/packages/arrow)
+> library is a format codec (IPC encode/decode on the BEAM heap, no NIFs).
+> ExArrow is the native zero-copy platform (Parquet, Flight, Flight SQL, ADBC,
+> pipelines). Both speak IPC binaries and can interoperate.
 
 ---
 
@@ -204,7 +212,7 @@ Add the dependency:
 
 ```elixir
 def deps do
-  [{:ex_arrow, "~> 0.7"}]
+  [{:ex_arrow, "~> 0.8"}]
 end
 ```
 
@@ -232,11 +240,11 @@ For **path dependencies** in Livebook (`Mix.install`), open notebooks from
 is detected) or use the Hex package:
 
 ```elixir
-Mix.install([{:ex_arrow, "~> 0.7.2"}, {:rustler, "~> 0.36", optional: true}])
+Mix.install([{:ex_arrow, "~> 0.8.0"}, {:rustler, "~> 0.36", optional: true}])
 ```
 
 Alternatively, use the published Hex package so the precompiled NIF is used
-and no Rust is needed: `Mix.install([{:ex_arrow, "~> 0.7.2"}])`.
+and no Rust is needed: `Mix.install([{:ex_arrow, "~> 0.8.0"}])`.
 
 ---
 
@@ -395,8 +403,9 @@ Interactive notebooks (open in [Livebook](https://livebook.dev)):
 - **[02 Flight](livebook/02_flight.livemd)** — Echo server, client, metadata APIs, Flight SQL prepared statements.
 - **[03 ADBC](livebook/03_adbc.livemd)** — Database, Connection, Statement, Stream (`:adbc_package` in Livebook).
 - **[04 ADBC integration](livebook/04_adbc_integration.livemd)** — Connection pooling with NimblePool.
+- **[05 Parquet](livebook/05_parquet.livemd)** — Pushdown reads, compressed writes, multi-file directories, PyArrow side-by-side.
 
-See [livebook/README.md](livebook/README.md) for run instructions.  Notebooks use Hex `~> 0.7.2` by default; opening from `livebook/` in a clone builds from source.
+See [livebook/README.md](livebook/README.md) for run instructions.  Notebooks use Hex `~> 0.8.0` by default; opening from `livebook/` in a clone builds from source.
 
 ---
 
@@ -682,35 +691,44 @@ package is available: `ExArrow.ADBC.DriverHelper.ensure_driver_and_open/2`.
 
 ## Parquet: read and write
 
-**Read from file:**
+**Read with pushdown (v0.8+):**
 
 ```elixir
-{:ok, stream}  = ExArrow.Parquet.Reader.from_file("/data/events.parquet")
-{:ok, schema}  = ExArrow.Stream.schema(stream)
-batches = ExArrow.Stream.to_list(stream)
+{:ok, stream} =
+  ExArrow.Stream.from_parquet("/data/events.parquet",
+    columns: ["user_id", "score"],
+    filters: {:gt, "score", 0.9}
+  )
+
+ExArrow.Parquet.Reader.read_stats(stream)
+# %{row_groups_total: ..., row_groups_selected: ..., row_groups_skipped: ...}
 ```
 
-**Read from binary (e.g. downloaded from S3):**
+**Multi-file / directory:**
 
 ```elixir
-{:ok, stream} = ExArrow.Parquet.Reader.from_binary(parquet_bytes)
-batch = ExArrow.Stream.next(stream)
+{:ok, stream} = ExArrow.Stream.from_parquet_dir("/data/events/")
 ```
 
-**Write to file:**
+**Write with compression:**
 
 ```elixir
-:ok = ExArrow.Parquet.Writer.to_file("/out/result.parquet", schema, batches)
+:ok =
+  ExArrow.Parquet.Writer.to_file("/out/result.parquet", schema, batches,
+    compression: :zstd,
+    row_group_size: 64_000
+  )
 ```
 
-**Write to binary (e.g. upload to object storage):**
+**Footer metadata (no row decode):**
 
 ```elixir
-{:ok, parquet_bytes} = ExArrow.Parquet.Writer.to_binary(schema, batches)
+{:ok, meta} = ExArrow.Parquet.Metadata.from_file("/data/events.parquet")
+meta.num_row_groups
 ```
 
-Parquet streams share the same `ExArrow.Stream` interface as IPC and ADBC
-streams — `schema/1`, `next/1`, and `to_list/1` all work identically.
+Full guide: [docs/parquet_guide.md](docs/parquet_guide.md). Livebook:
+`livebook/05_parquet.livemd`.
 
 ---
 

bench/parquet_pushdown_bench.exs

@@ -0,0 +1,40 @@
+# Pushdown vs post-read filter — rough timing helper for announcements.
+# Usage: mix run bench/parquet_pushdown_bench.exs
+
+alias ExArrow.Parquet
+alias ExArrow.RecordBatch
+alias ExArrow.Stream
+
+rows = 100_000
+ids = for i <- 1..rows, into: <<>>, do: <<i::little-signed-64>>
+scores = for i <- 1..rows, into: <<>>, do: <<:math.fmod(i * 0.01, 1.0)::little-float-64>>
+
+{:ok, batch} = RecordBatch.from_columns(["id", "score"], [ids, scores], ["s64", "f64"], rows)
+schema = RecordBatch.schema(batch)
+path = Path.join(System.tmp_dir!(), "ex_arrow_pushdown_bench.parquet")
+:ok = Parquet.Writer.to_file(path, schema, [batch], compression: :zstd, row_group_size: 10_000)
+
+measure = fn label, fun ->
+  {us, result} = :timer.tc(fun)
+  IO.puts("#{label}: #{Float.round(us / 1000, 1)} ms -> #{inspect(result)}")
+end
+
+measure.("pushdown filter", fn ->
+  {:ok, s} = Stream.from_parquet(path, columns: ["id"], filters: {:gt, "score", 0.9})
+  Enum.sum(Enum.map(Stream.to_list(s), &RecordBatch.num_rows/1))
+end)
+
+measure.("full read + project (no filter — no public scalar-predicate kernel yet)", fn ->
+  {:ok, s} = Stream.from_parquet(path)
+  s
+  |> Stream.to_list()
+  |> Enum.map(fn b ->
+    {:ok, projected} = ExArrow.Compute.project(b, ["score"])
+    RecordBatch.num_rows(projected)
+  end)
+  |> Enum.sum()
+end)
+
+IO.puts("stats after pushdown open:")
+{:ok, s} = Stream.from_parquet(path, filters: {:gt, "score", 0.9})
+IO.inspect(Parquet.Reader.read_stats(s))

docs/benchmarks.md

@@ -117,6 +117,20 @@ Compares Arrow `RecordBatch` against `list(map())` for build, transform
 (column select), and drain (sum a column) at 1K / 100K / 1M rows.  Quantifies
 the cost of going through row maps versus staying in native Arrow memory.
 
+### v0.8.0 Parquet pushdown (`bench/parquet_pushdown_bench.exs`)
+
+A standalone rough-timing helper (uses `:timer.tc/1`, not Benchee) that
+compares reading a 100K-row Parquet file with a pushdown filter against a
+full read plus post-read projection (no scalar-predicate kernel exists yet
+for an equivalent post-filter).  Run it directly:
+
+```bash
+MIX_ENV=dev mix run bench/parquet_pushdown_bench.exs
+```
+
+It prints elapsed milliseconds and the row-group selection stats after open.
+Not part of `bench/run_all.exs` (no HTML/JSON output).
+
 ## Published results
 
 Benchmark results from every push to `main` are stored in the `gh-pages`

docs/overview.md

@@ -3,6 +3,34 @@
 The main overview, installation, quick start, and usage examples live in the
 [README on GitHub](https://github.com/thanos/ex_arrow/blob/main/README.md).
 
+## What's changed in v0.8.0
+
+v0.8.0 makes Parquet a first-class citizen for larger-than-memory workloads.
+Read pushdown, write options, footer metadata, and multi-file directory
+streams let you scan only the data you need.
+
+New / changed:
+
+- **Read pushdown** — `ExArrow.Stream.from_parquet/2` and
+  `Parquet.Reader.from_file/2` / `from_binary/2` accept `:columns`
+  (projection), `:row_groups` (selection), and `:filters` (a predicate AST
+  with row-group statistics pruning). `Parquet.Reader.read_stats/1` reports
+  selected vs skipped row groups.
+- **Write options** — `:compression` (`:snappy`, `:zstd`, `{:zstd, level}`,
+  `:lz4`, `:gzip`, `:none`, alias `:uncompressed`), `:row_group_size`, and `:dictionary` on
+  `Writer.to_file/4` and `to_binary/3`.
+- **`ExArrow.Parquet.Metadata`** — footer-only metadata (`from_file/1`,
+  `from_binary/1`): row groups, per-column statistics, key-value metadata.
+- **Multi-file streams** — `ExArrow.Stream.from_parquet_files/2` and
+  `from_parquet_dir/2` lazily open one file at a time with schema checking;
+  early `Enum.take/2` does not open later files.
+- **`ExArrow.IPC.File.write/3`** and **`ExArrow.RecordBatch.concat/1`** —
+  public wrappers over existing NIFs.
+- Optional `arrow_testing` IPC conformance suite and a Mix.install
+  optional-dep smoke CI job.
+
+See the [Parquet guide](parquet_guide.md) and `livebook/05_parquet.livemd`.
+
 ## v0.7.2 stability update
 
 v0.7.2 fixes optional `:gen_stage` compilation in `Mix.install` and cached-build