Skip to content

Commit e78ba0d

Browse files
authored
Port lance-context 0.6.5 to riscv64 (#2726)
maturin/pyo3 build, abi3-py39 unconditional (one cp39-abi3 wheel). lance-core/lance-linalg (pulled in transitively via lance = "7.0.0") have no riscv64 SIMD fallback, same gap already patched for pylance/ lancedb/rerun-sdk; vendor riscv64-fixed crates.io copies and redirect via [patch.crates-io]. Also drops the unused lance-graph dependency, whose own 1.x pin of the same two crates can't be patched alongside the 7.0.0 one Cargo's [patch] table allows only one redirect per name.
1 parent 7358ec3 commit e78ba0d

6 files changed

Lines changed: 1807 additions & 0 deletions
Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
# SPDX-FileCopyrightText: 2026 The RISE Project
2+
# SPDX-License-Identifier: MIT
3+
---
4+
# This workflow is based on the `build`/`linux-x86_64` leg of upstream's own
5+
# wheel pipeline:
6+
# https://github.com/lance-format/lance-context/blob/v0.6.5/.github/workflows/python-publish.yml
7+
name: Build lance-context wheels (riscv64)
8+
9+
on:
10+
workflow_dispatch:
11+
inputs:
12+
version:
13+
description: 'Version glob to (re)build; empty builds every version of docs/packages/lance-context.yaml not released yet'
14+
required: false
15+
default: ''
16+
pull_request:
17+
branches: [main]
18+
paths:
19+
- '.github/workflows/build-lance-context.yml'
20+
- 'docs/packages/lance-context.yaml'
21+
- 'patches/lance-context/**'
22+
push:
23+
branches: [main]
24+
paths:
25+
- '.github/workflows/build-lance-context.yml'
26+
- 'docs/packages/lance-context.yaml'
27+
- 'patches/lance-context/**'
28+
29+
concurrency:
30+
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
31+
cancel-in-progress: true
32+
33+
permissions:
34+
contents: read # to fetch code (actions/checkout)
35+
36+
env:
37+
MANYLINUX_RISCV64_IMAGE: quay.io/pypa/manylinux_2_39_riscv64
38+
39+
jobs:
40+
setup:
41+
uses: $/.github/workflows/_setup.yml
42+
with:
43+
package: lance-context
44+
version: ${{ inputs.version }}
45+
46+
build_wheels:
47+
needs: [setup]
48+
if: needs.setup.outputs.versions != '[]'
49+
strategy:
50+
fail-fast: false
51+
matrix:
52+
version: ${{ fromJSON(needs.setup.outputs.versions) }}
53+
# pyo3 carries `abi3-py39` unconditionally in python/Cargo.toml, so
54+
# upstream ships one cp39-abi3 wheel. Our registry's pyarrow (a hard
55+
# runtime dependency) has no riscv64 wheel below cp312, so the build
56+
# floor here is cp312 instead -- the wheel is still tagged cp39-abi3 by
57+
# the pyo3 feature regardless of which interpreter compiles it (CLAUDE.md
58+
# gotcha 96), it is simply only tested on cp312 (same as
59+
# build-pylance.yml/build-lancedb.yml). python/pyproject.toml's own
60+
# `requires-python = ">=3.11,<3.14"` would also rule out cp314/cp314t,
61+
# had the pyarrow floor not already stopped at one interpreter.
62+
name: Build lance-context ${{ matrix.version }} cp39-abi3-manylinux_riscv64
63+
runs-on: ubuntu-24.04-riscv
64+
timeout-minutes: 1440
65+
66+
env:
67+
LANCE_CONTEXT_VERSION: ${{ matrix.version }}
68+
69+
steps:
70+
# python/ is a member of the lance-context Cargo workspace and builds
71+
# against its sibling crates, so the checkout lands at the workspace root.
72+
- name: Checkout lance-context v${{ env.LANCE_CONTEXT_VERSION }}
73+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
74+
with:
75+
repository: lance-format/lance-context
76+
ref: v${{ env.LANCE_CONTEXT_VERSION }}
77+
persist-credentials: false
78+
79+
- name: Checkout python-wheels
80+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
81+
with:
82+
path: python-wheels
83+
sparse-checkout: patches/lance-context
84+
persist-credentials: false
85+
86+
# lance-graph is declared in crates/lance-context-core/Cargo.toml but
87+
# never imported anywhere in this workspace, and its own Cargo.toml
88+
# pins lance-core/lance-linalg at an incompatible `1.x` line that
89+
# Cargo's [patch] table cannot redirect alongside the `7.0.0` line
90+
# crates/lance-context-core depends on directly (CLAUDE.md gotcha 300).
91+
- name: Apply lance-context patches
92+
run: git apply -v python-wheels/patches/lance-context/${{ env.LANCE_CONTEXT_VERSION }}/0001*.patch
93+
94+
# lance-core's SIMD_SUPPORT static and lance-linalg's f32x8/f32x16/
95+
# i32x8/f64x4/f64x8 SIMD types compile only for x86_64/aarch64/
96+
# loongarch64 upstream (CLAUDE.md gotcha 300); both are pulled in from
97+
# crates.io at a pinned `lance = "7.0.0"` (crates/lance-context-core/
98+
# Cargo.toml), so riscv64-fixed copies are vendored here and wired in
99+
# via `[patch.crates-io]`.
100+
- name: Vendor riscv64-fixed lance-core and lance-linalg
101+
run: |
102+
curl -fsSL -A "python-wheels (https://github.com/riseproject-dev/python-wheels)" -o lance-core.tar.gz "https://crates.io/api/v1/crates/lance-core/7.0.0/download"
103+
mkdir lance-core-7.0.0-riscv64
104+
tar xzf lance-core.tar.gz -C lance-core-7.0.0-riscv64 --strip-components=1
105+
rm lance-core.tar.gz
106+
patch -p1 -d lance-core-7.0.0-riscv64 < python-wheels/patches/lance-context/${{ env.LANCE_CONTEXT_VERSION }}/0002-lance-core-riscv64-simd-fallback.patch
107+
108+
curl -fsSL -A "python-wheels (https://github.com/riseproject-dev/python-wheels)" -o lance-linalg.tar.gz "https://crates.io/api/v1/crates/lance-linalg/7.0.0/download"
109+
mkdir lance-linalg-7.0.0-riscv64
110+
tar xzf lance-linalg.tar.gz -C lance-linalg-7.0.0-riscv64 --strip-components=1
111+
rm lance-linalg.tar.gz
112+
patch -p1 -d lance-linalg-7.0.0-riscv64 < python-wheels/patches/lance-context/${{ env.LANCE_CONTEXT_VERSION }}/0004-lance-linalg-riscv64-simd-fallback.patch
113+
114+
git apply python-wheels/patches/lance-context/${{ env.LANCE_CONTEXT_VERSION }}/0003-cargo-patch-lance-crates-for-riscv64.patch
115+
116+
- name: Free disk space
117+
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
118+
119+
# This workspace links ~960 crates (datafusion, arrow, the embedded
120+
# lancedb/lance/lance-index Rust crates, the aws/azure/gcp/huggingface/
121+
# tencent object-store backends, ...); same OOM guard build-lancedb.yml/
122+
# build-rerun-sdk.yml need for a comparably sized Rust workspace.
123+
- name: Set swap space
124+
uses: pierotofy/set-swap-space@fc79b3f67fa8a838184ce84a674ca12238d2c761 # master
125+
with:
126+
swap-size-gb: 16
127+
128+
- name: Build wheel
129+
uses: pypa/cibuildwheel@1828c10ab37f080699c7b81cea34097c684a7074 # v4.2.0
130+
with:
131+
package-dir: python
132+
output-dir: wheelhouse/
133+
# musllinux is dropped: rustup.rs ships no riscv64 musl toolchain.
134+
only: cp312-manylinux_riscv64
135+
env:
136+
CIBW_MANYLINUX_RISCV64_IMAGE: ${{ env.MANYLINUX_RISCV64_IMAGE }}
137+
# lance-context ships no [tool.cibuildwheel], so the Rust toolchain
138+
# its maturin backend needs is installed in-container here. Rocky
139+
# 10's CRB repo (enabled in the manylinux image) has protoc;
140+
# PROTOC_INCLUDE is needed because it resolves
141+
# google/protobuf/*.proto from disk rather than from the binary
142+
# (same as build-pylance.yml/build-lancedb.yml).
143+
CIBW_BEFORE_ALL_LINUX: >-
144+
yum install -y protobuf-compiler protobuf-devel &&
145+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
146+
# python/pyproject.toml leaves [tool.maturin] at the default `dev`
147+
# profile, so a plain PEP 517 build ships an unoptimised wheel
148+
# unless `--profile release` is passed explicitly (CLAUDE.md
149+
# gotcha 141; the deltalake case).
150+
CIBW_ENVIRONMENT_LINUX: >-
151+
PATH="$PATH:$HOME/.cargo/bin"
152+
PROTOC=/usr/bin/protoc
153+
PROTOC_INCLUDE=/usr/include
154+
MATURIN_PEP517_ARGS="--profile release --strip"
155+
CARGO_BUILD_JOBS=2
156+
PIP_EXTRA_INDEX_URL=https://pypi.riseproject.dev/simple/
157+
# Without this pip prefers PyPI's pyarrow, which has no riscv64
158+
# wheel, and source-builds Arrow's whole C++ library here.
159+
CIBW_TEST_ENVIRONMENT: PIP_ONLY_BINARY=pyarrow
160+
# The `tests` extra also lists pylance/lance-namespace/lancedb, but
161+
# none of this suite imports them (only `pyarrow` and stub
162+
# embedding providers -- see test_embeddings.py/
163+
# test_multimodal_embeddings.py); pinning the extra as-is would
164+
# resolve pylance>=7,<8/lancedb==0.27.1 against versions our
165+
# registry doesn't carry (it only hosts pylance==11.0.0/
166+
# lancedb==0.37.1+) and either fail or source-build the whole lance
167+
# Rust workspace a second time, so the suite's actual requirements
168+
# are listed directly instead.
169+
CIBW_TEST_REQUIRES: pytest pytest-asyncio moto[s3,server] boto3 botocore
170+
CIBW_TEST_SOURCES: python/pyproject.toml python/tests
171+
# Mirrors upstream's own `test`/`Run tests` step (python-test.yml:
172+
# `pytest -v` from python/, no extra flags). test_rollout_remote.py
173+
# skips itself when target/debug/lance-context-server hasn't been
174+
# built, which it isn't here (that crate is a workspace member
175+
# neither the Python wheel nor its tests otherwise need).
176+
CIBW_TEST_COMMAND: >-
177+
cd python &&
178+
python -m pytest tests -v
179+
180+
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
181+
with:
182+
name: lance-context-${{ env.LANCE_CONTEXT_VERSION }}-cp39-abi3-manylinux_riscv64
183+
path: wheelhouse/*.whl
184+
if-no-files-found: error
185+
186+
publish:
187+
name: Publish lance-context ${{ matrix.version }}
188+
needs: [setup, build_wheels]
189+
if: needs.setup.outputs.versions != '[]'
190+
strategy:
191+
fail-fast: false
192+
matrix:
193+
version: ${{ fromJSON(needs.setup.outputs.versions) }}
194+
permissions:
195+
contents: write
196+
pull-requests: write
197+
uses: $/.github/workflows/_publish-wheel.yml
198+
secrets:
199+
app-private-key: ${{ secrets.RISEPROJECT_APP_PRIVATE_KEY }}
200+
with:
201+
artifact-pattern: lance-context-${{ matrix.version }}-*-manylinux_riscv64

‎docs/packages/lance-context.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
package-name: lance-context
2+
source-code: https://github.com/lance-format/lance-context
3+
license: Apache-2.0
4+
versions:
5+
- version: 0.6.5
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
2+
From: Ludovic Henry <git@ludovic.dev>
3+
Date: Mon, 06 Oct 2026 00:00:00 +0000
4+
Subject: [PATCH] lance-context-core: drop the unused lance-graph dependency
5+
6+
crates/lance-context-core/Cargo.toml depends on `lance-graph = "0.5.4"`,
7+
but no Rust source file in this workspace imports `lance_graph`
8+
(`grep -rln lance_graph crates/ python/` matches nothing); it is dead
9+
weight in the dependency graph. lance-graph's own Cargo.toml pins
10+
`lance`/`lance-core`/`lance-linalg` at an incompatible `1.x` line, which
11+
Cargo resolves as a second, parallel copy of the whole lance-core/
12+
lance-linalg family alongside the `7.0.0` one `crates/lance-context-core`
13+
depends on directly (`lance = "7.0.0"`). That `1.x` copy's lance-core
14+
SIMD_SUPPORT closure and lance-linalg's f32x8/f32x16/i32x8/f64x4/f64x8
15+
SIMD types have no riscv64 fallback either, and Cargo's `[patch]` table
16+
can only redirect one semver-incompatible version of a given crate name
17+
at a time (verified empirically: patching crates-io `foo` to a local
18+
`1.0.0` copy while another dependency requires `foo = "2"` fails
19+
resolution with "candidate versions found which didn't match"), so
20+
patching lance-core/lance-linalg 7.0.0 for riscv64 (see
21+
0003-cargo-patch-lance-crates-for-riscv64.patch and
22+
0002/0004-lance-{core,linalg}-riscv64-simd-fallback.patch) cannot also
23+
cover the unused 1.x copy lance-graph alone drags in. Removing the dead
24+
dependency drops that whole second copy from resolution instead.
25+
26+
Upstream-Status: To upstream [not yet submitted to lance-format/lance-context]
27+
28+
Signed-off-by: Ludovic Henry <git@ludovic.dev>
29+
---
30+
crates/lance-context-core/Cargo.toml | 1 -
31+
1 file changed, 1 deletion(-)
32+
33+
diff --git a/crates/lance-context-core/Cargo.toml b/crates/lance-context-core/Cargo.toml
34+
index c47d8c8..98dda4c 100644
35+
--- a/crates/lance-context-core/Cargo.toml
36+
+++ b/crates/lance-context-core/Cargo.toml
37+
@@ -30,7 +30,6 @@ lance-context-api = { version = "0.6.5", path = "../lance-context-api" }
38+
lance-index = "7.0.0"
39+
lance-namespace = "7.0.0"
40+
lancedb = "0.30.0"
41+
-lance-graph = "0.5.4"
42+
# Version-matched with lance-context-server/-master so one process-wide recorder
43+
# serves every crate.
44+
metrics = { version = "0.24", optional = true }
45+
--
46+
2.43.0
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
2+
From: Ludovic Henry <git@ludovic.dev>
3+
Date: Mon, 06 Oct 2026 00:00:00 +0000
4+
Subject: [PATCH] lance-core: add a portable riscv64 SIMD-tier fallback
5+
6+
lance-core's SIMD_SUPPORT static (src/utils/cpu.rs) picks a SIMD tier
7+
via a `cfg`-gated closure with arms for aarch64, x86_64 and loongarch64
8+
only; none of them match riscv64, so the closure's block has no
9+
trailing expression there and the crate fails to compile with "expected
10+
`SimdSupport`, found `()`". Add a `None`-tier fallback arm beside the
11+
existing ones for every architecture without a dedicated tier.
12+
13+
Applied (via patches/lance-context/0.6.5/0003-cargo-patch-lance-crates-for-riscv64.patch)
14+
to a local copy of this crate's 7.0.0 crates.io release, which
15+
crates/lance-context-core's `lance = "7.0.0"` pulls in transitively
16+
and cannot be patched in place (CLAUDE.md gotcha 300).
17+
18+
Upstream-Status: To upstream [not yet submitted to lance-format/lance]
19+
20+
Signed-off-by: Ludovic Henry <git@ludovic.dev>
21+
---
22+
diff --git a/src/utils/cpu.rs b/src/utils/cpu.rs
23+
index 4e7ab01..34ed3a0 100644
24+
--- a/src/utils/cpu.rs
25+
+++ b/src/utils/cpu.rs
26+
@@ -58,6 +58,17 @@ pub static SIMD_SUPPORT: LazyLock<SimdSupport> = LazyLock::new(|| {
27+
SimdSupport::None
28+
}
29+
}
30+
+ // riscv64 (and any other architecture without a dedicated tier above) has
31+
+ // no SIMD kernels in lance yet; upstream's cfg arms have no catch-all, so
32+
+ // this closure otherwise evaluates to `()` here instead of `SimdSupport`.
33+
+ #[cfg(not(any(
34+
+ target_arch = "aarch64",
35+
+ target_arch = "x86_64",
36+
+ target_arch = "loongarch64"
37+
+ )))]
38+
+ {
39+
+ SimdSupport::None
40+
+ }
41+
});
42+
43+
#[cfg(target_arch = "x86_64")]
44+
--
45+
2.43.0
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
2+
From: Ludovic Henry <git@ludovic.dev>
3+
Date: Mon, 06 Oct 2026 00:00:00 +0000
4+
Subject: [PATCH] cargo: redirect lance-core/lance-linalg to a riscv64-patched
5+
checkout
6+
7+
lance-core's SIMD_SUPPORT static (src/utils/cpu.rs) and lance-linalg's
8+
f32x8/f32x16/i32x8/f64x4/f64x8 SIMD types (src/simd/{f32,i32,f64}.rs)
9+
are compiled for x86_64/aarch64/loongarch64 only, with no fallback arm
10+
for any other architecture (CLAUDE.md gotcha 300): the SIMD_SUPPORT
11+
closure has no trailing expression on riscv64 ("expected SimdSupport,
12+
found ()"), and the SIMD type structs are defined only under those
13+
three `#[cfg(target_arch = ...)]` attributes while every method
14+
implementing them is unconditional, so riscv64 hits "cannot find type
15+
`f32x8` in this scope" as soon as lance-linalg is built.
16+
17+
Both crates are pulled in from crates.io at a pinned `=7.0.0` via
18+
`crates/lance-context-core`'s `lance = "7.0.0"` (lance-graph's own,
19+
unused `1.x` copy of the same two crates is dropped entirely by
20+
0001-remove-unused-lance-graph-dependency.patch instead -- Cargo's
21+
`[patch]` table can only redirect one semver-incompatible version of a
22+
given crate name at a time), so the fix cannot be applied in place.
23+
Redirect them to the riscv64-patched copies the workflow materializes
24+
at build time from the pinned crates.io release (see
25+
0002-lance-core-riscv64-simd-fallback.patch and
26+
0004-lance-linalg-riscv64-simd-fallback.patch), at the same version
27+
Cargo.lock already pins -- only the `source`/`checksum` lines of the
28+
lance-core/lance-linalg 7.0.0 entries change there, so every other
29+
dependency stays pinned exactly as upstream released it.
30+
31+
Upstream-Status: Inappropriate [redirects a pinned dependency to a local checkout materialized at build time; the fix itself belongs in lance-format/lance]
32+
33+
Signed-off-by: Ludovic Henry <git@ludovic.dev>
34+
---
35+
diff --git a/Cargo.lock b/Cargo.lock
36+
index 3a467b8..83bb9e5 100644
37+
--- a/Cargo.lock
38+
+++ b/Cargo.lock
39+
@@ -5625,8 +5625,6 @@ dependencies = [
40+
[[package]]
41+
name = "lance-core"
42+
version = "7.0.0"
43+
-source = "registry+https://github.com/rust-lang/crates.io-index"
44+
-checksum = "13f84020da5a484e2f07dd1796e09785ed7cd889857ebc4cb77e32ef214ee594"
45+
dependencies = [
46+
"arrow-array 58.3.0",
47+
"arrow-buffer 58.3.0",
48+
@@ -6219,8 +6217,6 @@ dependencies = [
49+
[[package]]
50+
name = "lance-linalg"
51+
version = "7.0.0"
52+
-source = "registry+https://github.com/rust-lang/crates.io-index"
53+
-checksum = "6b4c51cad0ac780b02dc4da48528479e7693c03e8d05390510bbc69ca2a9a1f1"
54+
dependencies = [
55+
"arrow-array 58.3.0",
56+
"arrow-buffer 58.3.0",
57+
diff --git a/Cargo.toml b/Cargo.toml
58+
index 2efab19..618ba8b 100644
59+
--- a/Cargo.toml
60+
+++ b/Cargo.toml
61+
@@ -17,3 +17,12 @@ license = "Apache-2.0"
62+
63+
[workspace.metadata]
64+
# reserved for future use
65+
+
66+
+# lance-core's SIMD_SUPPORT static and lance-linalg's f32x8/f32x16/i32x8/
67+
+# f64x4/f64x8 SIMD types compile only for x86_64/aarch64/loongarch64
68+
+# upstream (CLAUDE.md gotcha 300); the riscv64-patched copies below are
69+
+# materialized from the pinned crates.io release by the workflow at
70+
+# build time.
71+
+[patch.crates-io]
72+
+lance-core = { path = "lance-core-7.0.0-riscv64" }
73+
+lance-linalg = { path = "lance-linalg-7.0.0-riscv64" }
74+
--
75+
2.43.0

0 commit comments

Comments
 (0)