Skip to content

Commit 34ee386

Browse files
authored
runai-model-streamer-s3: Add version 0.16.1 (#2355)
1 parent fb308f9 commit 34ee386

7 files changed

Lines changed: 805 additions & 0 deletions

‎.github/workflows/build-runai-model-streamer-s3.yml‎

Lines changed: 484 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
package-name: runai-model-streamer-s3
2+
source-code: https://github.com/run-ai/runai-model-streamer
3+
license: Apache-2.0 AND curl AND Zlib
4+
versions:
5+
- version: 0.16.1
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
From 820707245ffc7f9f363fc5de3f5375daacb0a8b9 Mon Sep 17 00:00:00 2001
2+
From: Ludovic Henry <git@ludovic.dev>
3+
Date: Thu, 24 Sep 2026 03:32:40 +0000
4+
Subject: [PATCH] toolchain: add riscv64 as a supported architecture
5+
6+
configure_toolchain() and its per-arch .bazelrc block are already fully
7+
generic over the arch string (native.platform constraint is
8+
@platforms//cpu:<arch>, gcc tools are resolved as
9+
/usr/bin/<arch>-linux-gnu-<tool>) -- riscv64 was simply never added to
10+
either list.
11+
12+
No -march flag is added for riscv64 (unlike aarch64's, which exists only
13+
for a google-cloud-cpp ARM dependency libstreamer.so does not build
14+
against): a native riscv64 build uses the compiler's own default target.
15+
16+
Upstream-Status: To upstream [this port does not open issues/PRs against third-party repos; riscv64 toolchain support is exactly the kind of change upstream would want once a downstream build proves it works]
17+
---
18+
cpp/.bazelrc | 3 +++
19+
cpp/toolchain/configure.bzl | 2 +-
20+
2 files changed, 4 insertions(+), 1 deletion(-)
21+
22+
diff --git a/cpp/.bazelrc b/cpp/.bazelrc
23+
index 830e568..4a6ccd2 100644
24+
--- a/cpp/.bazelrc
25+
+++ b/cpp/.bazelrc
26+
@@ -19,5 +19,8 @@ build:aarch64 --copt="-march=armv8-a+crc+crypto"
27+
build:x86_64 --platform_suffix=x86_64
28+
build:x86_64 --platforms=@toolchain//:x86_64
29+
30+
+build:riscv64 --platform_suffix=riscv64
31+
+build:riscv64 --platforms=@toolchain//:riscv64
32+
+
33+
# Azurite testing support (enables AZURE_STORAGE_ACCOUNT_KEY for local testing)
34+
build:azurite --define azurite_testing=true
35+
diff --git a/cpp/toolchain/configure.bzl b/cpp/toolchain/configure.bzl
36+
index 6b89f54..9a04125 100644
37+
--- a/cpp/toolchain/configure.bzl
38+
+++ b/cpp/toolchain/configure.bzl
39+
@@ -4,7 +4,7 @@ load("//toolchain:rules.bzl", "get_target_triplet", "runai_crosstool_tools")
40+
41+
# This should map to the devcontainer Dockerfile
42+
# We install gcc-x86-64-linux-gnu and gcc-aarch64-linux-gnu toolchains
43+
-ARCHITECTURES = ["x86_64", "aarch64"]
44+
+ARCHITECTURES = ["x86_64", "aarch64", "riscv64"]
45+
OS = "linux-gnu"
46+
47+
def _get_gcc_version(repository_ctx, arch):
48+
--
49+
2.43.0
50+
Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
From 75c45793987328b8011252f0f24950e94456b336 Mon Sep 17 00:00:00 2001
2+
From: Ludovic Henry <git@ludovic.dev>
3+
Date: Thu, 24 Sep 2026 08:01:06 +0000
4+
Subject: [PATCH] toolchain: use gcc's own -dumpmachine triplet for builtin
5+
include dirs
6+
7+
configure_toolchain()'s cxx_builtin_include_directories are built from the
8+
same os+arch naming-convention triplet (<arch>-linux-gnu) used to resolve
9+
tool paths (/usr/bin/<arch>-linux-gnu-<tool>). That equivalence holds for a
10+
genuine Debian cross-toolchain package (a real x86_64-linux-gnu-gcc reports
11+
exactly that as its own -dumpmachine), but not when the tool path is a
12+
same-arch *native* compiler reachable only via a symlink under the expected
13+
name -- its own compiled-in builtin search path (/usr/lib/gcc/<real
14+
triplet>/<ver>/include, etc.) still uses its real vendor triplet regardless
15+
of what name it is invoked under.
16+
17+
This is exactly riscv64's manylinux_riscv64 image: gcc is Red Hat's native
18+
riscv64-redhat-linux build, symlinked to /usr/bin/riscv64-linux-gnu-gcc so
19+
runai_crosstool_tools() finds it. Bazel's own absolute-path sandboxing then
20+
rejects gcc's builtin headers (e.g. stddef.h) because
21+
/usr/lib/gcc/riscv64-redhat-linux/<ver>/include is not in the declared
22+
riscv64-linux-gnu-based list.
23+
24+
Thread the compiler's actual -dumpmachine output down as gcc_machine and use
25+
it (falling back to the naming-convention triplet when unset, so x86_64 and
26+
aarch64 are unaffected) for cxx_builtin_include_directories specifically,
27+
while tool_paths keeps using the naming-convention triplet the workflow's
28+
symlinks provide.
29+
30+
Upstream-Status: To upstream [this port does not open issues/PRs against third-party repos; making cxx_builtin_include_directories match the compiler's own reported triplet is a general robustness fix upstream would likely want]
31+
---
32+
cpp/toolchain/configure.bzl | 7 ++++++-
33+
cpp/toolchain/template/toolchain.bzl | 23 ++++++++++++++++++++---
34+
2 files changed, 26 insertions(+), 4 deletions(-)
35+
36+
diff --git a/cpp/toolchain/configure.bzl b/cpp/toolchain/configure.bzl
37+
index 9a04125..510ef90 100644
38+
--- a/cpp/toolchain/configure.bzl
39+
+++ b/cpp/toolchain/configure.bzl
40+
@@ -11,6 +11,10 @@ def _get_gcc_version(repository_ctx, arch):
41+
gcc_tool = runai_crosstool_tools(get_target_triplet(OS, arch))["gcc"]
42+
return repository_ctx.execute([gcc_tool, "-dumpversion"]).stdout.strip()
43+
44+
+def _get_gcc_machine(repository_ctx, arch):
45+
+ gcc_tool = runai_crosstool_tools(get_target_triplet(OS, arch))["gcc"]
46+
+ return repository_ctx.execute([gcc_tool, "-dumpmachine"]).stdout.strip()
47+
+
48+
def _get_host_arch(repository_ctx):
49+
return repository_ctx.execute(["/usr/bin/uname", "-m"]).stdout.strip()
50+
51+
@@ -18,9 +22,10 @@ def _cc_autoconf_toolchain_impl(repository_ctx):
52+
define_statements = []
53+
for arch in ARCHITECTURES:
54+
gcc_version = _get_gcc_version(repository_ctx, arch)
55+
+ gcc_machine = _get_gcc_machine(repository_ctx, arch)
56+
host_arch = _get_host_arch(repository_ctx)
57+
toolchain_name = arch
58+
- define_statements.append('define_toolchain(name = "%s", os = "%s", host_arch = "%s", arch = "%s", gcc_version = "%s")' % (toolchain_name, OS, host_arch, arch, gcc_version))
59+
+ define_statements.append('define_toolchain(name = "%s", os = "%s", host_arch = "%s", arch = "%s", gcc_version = "%s", gcc_machine = "%s")' % (toolchain_name, OS, host_arch, arch, gcc_version, gcc_machine))
60+
61+
repository_ctx.template(
62+
"BUILD",
63+
diff --git a/cpp/toolchain/template/toolchain.bzl b/cpp/toolchain/template/toolchain.bzl
64+
index 6a4c6cf..36daffc 100644
65+
--- a/cpp/toolchain/template/toolchain.bzl
66+
+++ b/cpp/toolchain/template/toolchain.bzl
67+
@@ -73,6 +73,15 @@ def _impl(ctx):
68+
target_triplet = get_target_triplet(ctx.attr.os, ctx.attr.arch)
69+
tool_paths = [tool_path(name = k, path = v) for k, v in runai_crosstool_tools(target_triplet).items()]
70+
71+
+ # gcc_machine is the triplet gcc itself reports (`gcc -dumpmachine`), which is
72+
+ # what its own builtin include search path is actually built from. It is
73+
+ # normally identical to target_triplet (a real x86_64-linux-gnu/aarch64-linux-gnu
74+
+ # cross-gcc reports exactly that), but differs when the "gcc" behind
75+
+ # target_triplet's tool path is a same-arch native compiler under a different
76+
+ # vendor triplet (e.g. Red Hat's riscv64-redhat-linux) made reachable only by a
77+
+ # symlink under the expected name.
78+
+ include_triplet = ctx.attr.gcc_machine if ctx.attr.gcc_machine else target_triplet
79+
+
80+
# Documented at
81+
# https://docs.bazel.build/versions/main/skylark/lib/cc_common.html#create_cc_toolchain_config_info.
82+
#
83+
@@ -89,7 +98,7 @@ def _impl(ctx):
84+
abi_version = "unknown",
85+
abi_libc_version = "unknown",
86+
tool_paths = tool_paths,
87+
- cxx_builtin_include_directories = _get_include_directories(target_triplet, ctx.attr.gcc_version, ctx.attr.use_cross),
88+
+ cxx_builtin_include_directories = _get_include_directories(include_triplet, ctx.attr.gcc_version, ctx.attr.use_cross),
89+
features = features
90+
)
91+
92+
@@ -111,6 +120,11 @@ _toolchain_config = rule(
93+
mandatory = True,
94+
doc = "GCC major version to use (eg: 9)",
95+
),
96+
+ "gcc_machine": attr.string(
97+
+ mandatory = False,
98+
+ default = "",
99+
+ doc = "The triplet `gcc -dumpmachine` reports, if it differs from os/arch's own naming convention",
100+
+ ),
101+
"use_cross": attr.bool(
102+
mandatory = False,
103+
default = False,
104+
@@ -121,7 +135,7 @@ _toolchain_config = rule(
105+
)
106+
107+
108+
-def define_toolchain(name, os, host_arch, arch, gcc_version):
109+
+def define_toolchain(name, os, host_arch, arch, gcc_version, gcc_machine = ""):
110+
""""define_toolchain creates rules for a cc_toolchain.
111+
112+
This expects toolchain tools to exist at a canonical path.
113+
@@ -134,6 +148,9 @@ def define_toolchain(name, os, host_arch, arch, gcc_version):
114+
host_arch: The host architecture for this toolchain
115+
arch: The target architecture for this toochain
116+
gcc_version: The GCC version of this toolchain
117+
+ gcc_machine: The triplet `gcc -dumpmachine` reports for this toolchain's
118+
+ compiler, if it differs from the os/arch naming convention used for
119+
+ the tool paths themselves
120+
"""
121+
122+
native.platform(
123+
@@ -145,7 +162,7 @@ def define_toolchain(name, os, host_arch, arch, gcc_version):
124+
)
125+
126+
toolchain_config_name = "%s_toolchain_config" % name
127+
- _toolchain_config(name = toolchain_config_name, os = os, arch = arch, gcc_version = gcc_version, use_cross = host_arch != arch)
128+
+ _toolchain_config(name = toolchain_config_name, os = os, arch = arch, gcc_version = gcc_version, gcc_machine = gcc_machine, use_cross = host_arch != arch)
129+
130+
empty_target_name = "%s_empty" % name
131+
empty_target_label = ":%s" % empty_target_name
132+
--
133+
2.43.0
134+
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
2+
From: Ludovic Henry <git@ludovic.dev>
3+
Date: Sat, 26 Sep 2026 06:00:00 +0000
4+
Subject: [PATCH] third_party: add riscv64 to the AWS SDK library select
5+
6+
The @aws alias selects a per-arch aws_library() (headers and static
7+
libraries under /opt/<arch>-aws, /opt/<arch>-ssl, ...) by target CPU, but
8+
only has x86_64 and aarch64 branches, so any other --config fails with
9+
"configurable attribute doesn't match this configuration". aws_library()
10+
itself is already generic over the arch string; add the matching
11+
config_setting and branch for riscv64.
12+
13+
Upstream-Status: To upstream [this port does not open issues/PRs against third-party repos; riscv64 support in the S3 backend is exactly the kind of change upstream would want once a downstream build proves it works]
14+
---
15+
cpp/third_party/aws.BUILD | 9 +++++++++
16+
1 file changed, 9 insertions(+)
17+
18+
diff --git a/cpp/third_party/aws.BUILD b/cpp/third_party/aws.BUILD
19+
index 7cab4ea..c81e4ac 100644
20+
--- a/cpp/third_party/aws.BUILD
21+
+++ b/cpp/third_party/aws.BUILD
22+
@@ -25,13 +25,22 @@ config_setting(
23+
],
24+
)
25+
26+
+config_setting(
27+
+ name = "target_riscv64",
28+
+ constraint_values = [
29+
+ "@platforms//cpu:riscv64",
30+
+ ],
31+
+)
32+
+
33+
aws_library(name = "aws_aarch", arch = "aarch64")
34+
aws_library(name = "aws_x86_64", arch = "x86_64")
35+
+aws_library(name = "aws_riscv64", arch = "riscv64")
36+
37+
alias(
38+
name = "aws",
39+
actual = select({
40+
":target_x86_64": ":aws_x86_64",
41+
":target_aarch64": ":aws_aarch",
42+
+ ":target_riscv64": ":aws_riscv64",
43+
}),
44+
)
45+
--
46+
2.43.0
47+
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
2+
From: Ludovic Henry <git@ludovic.dev>
3+
Date: Sat, 26 Sep 2026 06:00:00 +0000
4+
Subject: [PATCH] s3: ship the licences of the statically linked dependencies
5+
6+
libstreamers3.so statically links the AWS SDK for C++ (and its CRT
7+
libraries, s2n-tls included), OpenSSL, libcurl and zlib, but the explicit
8+
license_files=("LICENSE",) only ships the project's own licence. curl's
9+
licence and the Apache-2.0 NOTICE files of the AWS libraries require
10+
their notices to travel with the binary. Also match LICENSE.* next to
11+
setup.py, where the build stages each dependency's licence text.
12+
13+
Upstream-Status: To upstream [this port does not open issues/PRs against third-party repos; the upstream wheels have the same gap]
14+
---
15+
py/runai_model_streamer_s3/setup.py | 2 +-
16+
1 file changed, 1 insertion(+), 1 deletion(-)
17+
18+
diff --git a/py/runai_model_streamer_s3/setup.py b/py/runai_model_streamer_s3/setup.py
19+
index ae0e820..00c7b9a 100644
20+
--- a/py/runai_model_streamer_s3/setup.py
21+
+++ b/py/runai_model_streamer_s3/setup.py
22+
@@ -22,7 +22,7 @@ assert_lib_exists()
23+
setup(
24+
name="runai-model-streamer-s3",
25+
version=VERSION,
26+
- license_files=("LICENSE",),
27+
+ license_files=("LICENSE", "LICENSE.*"),
28+
packages=find_packages(),
29+
install_requires=["boto3"],
30+
data_files=[("/runai_model_streamer/libstreamer/lib/", [LIB])],
31+
--
32+
2.43.0
33+
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
2+
From: Ludovic Henry <git@ludovic.dev>
3+
Date: Sat, 26 Sep 2026 06:00:00 +0000
4+
Subject: [PATCH] s3: fix the stale _filter_allow/_filter_ignore unit tests
5+
6+
Since #80 the S3 package's test_filter_allow and
7+
test_filter_allow_full_path call _filter_ignore() with an allow_pattern
8+
keyword, and test_filter_ignore passes ignore_pattern, but both helpers
9+
in files/files.py take (paths, patterns). All three raise TypeError
10+
against a stock 0.16.1 install (PyPI's x86_64 wheel included); upstream
11+
CI never notices because py/Makefile's test target only runs the
12+
runai_model_streamer package's tests. Only the stale-test hunk of the
13+
upstream commit is taken, aligning with the GCS/Azure copies.
14+
15+
Upstream-Status: Backport [https://github.com/run-ai/runai-model-streamer/commit/c6a6543f195158ee3175b66504044dac1801de27]
16+
---
17+
.../runai_model_streamer_s3/files/tests/test_files.py | 10 +++++-----
18+
1 file changed, 5 insertions(+), 5 deletions(-)
19+
20+
diff --git a/py/runai_model_streamer_s3/runai_model_streamer_s3/files/tests/test_files.py b/py/runai_model_streamer_s3/runai_model_streamer_s3/files/tests/test_files.py
21+
index eb8ba87..0722c09 100644
22+
--- a/py/runai_model_streamer_s3/runai_model_streamer_s3/files/tests/test_files.py
23+
+++ b/py/runai_model_streamer_s3/runai_model_streamer_s3/files/tests/test_files.py
24+
@@ -9,23 +9,23 @@ import runai_model_streamer_s3.files.files as files
25+
26+
class TestFiles(unittest.TestCase):
27+
def test_filter_allow(self):
28+
- res = files._filter_ignore(
29+
+ res = files._filter_allow(
30+
["test_file1.txt1", "test_file2.txt2", "test_file3.txt3"],
31+
- allow_pattern=["*.txt2"]
32+
+ patterns=["*.txt2"]
33+
)
34+
self.assertEqual(res, ["test_file2.txt2"])
35+
36+
def test_filter_allow_full_path(self):
37+
- res = files._filter_ignore(
38+
+ res = files._filter_allow(
39+
["test_file1.txt1", "dir/test_file2.txt2", "test_file3.txt3"],
40+
- allow_pattern=["*.txt2"]
41+
+ patterns=["*.txt2"]
42+
)
43+
self.assertEqual(res, ["dir/test_file2.txt2"])
44+
45+
def test_filter_ignore(self):
46+
res = files._filter_ignore(
47+
["test_file1.txt1", "test_file2.txt2", "test_file3.txt3"],
48+
- ignore_pattern=["*.txt2"]
49+
+ patterns=["*.txt2"]
50+
)
51+
self.assertEqual(res, ["test_file1.txt1", "test_file3.txt3"])
52+

0 commit comments

Comments
 (0)