[build] Bazel 8.8.1: bump protobuf to v29.6, rules_java to 8.12.0 and gRPC to 1.69.0 by elliot-barn · Pull Request #66483 · ray-project/ray

elliot-barn · 2026-09-25T00:57:38Z

Why

This PR moves Ray to Bazel 8.8.1, still in WORKSPACE mode, together with the dependency bumps that Bazel 8 forces. It is the Bazel 8 step of the Bazel 7.5 → 8 → 9 migration. The earlier steps (#66160, #66207, #66305) got the tree as far as letting Bazel 8 start.

Bazel 8 needs rules_java 8.x. Every rules_java 8.x release loads @com_google_protobuf//bazel:java_proto_library.bzl, and that file does not exist in the v23.4-era protobuf commit Ray pins. So rules_java and protobuf have to move together, and protobuf ≥ 29 then pulls gRPC and abseil along.

Split for review

The parts that work on master on their own are split out into separate PRs. Please review those first. Once they merge, this PR is rebased and those commits drop out of the diff.

PR	What
#67717	Deflake four timing-sensitive tests that the gRPC bump made fail more often
#67718	A gRPC client with calls in flight is never idle (actor task ordering fix, see below)
#67719	Keep gRPC's absl logging at ERROR unless `GRPC_VERBOSITY` is set (see below)
#67720	`Arena::CreateMessage` → `Arena::Create`, and pin Java 8 bytecode ahead of `rules_java` 8
#67721	`bazel_skylib` 1.7.1 and `rules_pkg` 0.9.1

What stays here: the Bazel version, the protobuf / rules_java / gRPC / abseil / rules_cc bumps, the code and patch changes they need, the glibc 2.17 Bazel build for CI, and the Windows / sanitizer / test fixes for gRPC 1.69.

Versions

Repo	Before	After	Why it moves
Bazel	7.5.0	8.8.1	The goal. See "Bazel 8" below
`com_google_protobuf`	`2c5fa078` (~v23.4)	v29.6	`rules_java` 8.x loads from its `bazel/` package
`rules_java`	7.12.4	8.12.0	Needed for Bazel 8. 8.12.0 is the ceiling: from 8.13.0 on, the prebuilt `java_tools` (`singlejar`, `ijar`, `one_version`) need `GLIBCXX_3.4.21`, newer than manylinux2014's libstdc++. 8.x also changed the default Java language level from 8 to 11, so `.bazelrc` pins `--java_language_version=8` (#67720).
`com_github_grpc_grpc`	1.58.0	1.69.0	gRPC < 1.69 builds a standalone `@upb` that fails to compile against protobuf 29's `descriptor.proto` (`FileDescriptorProto.edition` is now an enum). 1.69.0 is the first gRPC whose `grpc_deps.bzl` pins protobuf 29.
`com_google_absl`	20230802.1	20240722.2	Matches gRPC 1.69.0. Patch release 2 fixes a security issue in hash container create/resize.
`rules_proto`	6.0.2 (from `grpc_deps()`)	7.1.0	6.0.2 reads native `proto_common.ProtoLangToolchainInfo`, which Bazel 8 removed
`rules_cc`	0.0.13 (transitive)	0.0.16	What protobuf 29 declares
`bazel_skylib`	1.6.1 (in name only)	1.7.1	Protobuf 29 calls `paths.is_normalized()` (#67721)
`rules_pkg`	0.7.0 (transitive)	0.9.1	Ceiling, see below (#67721)
`boringssl`	`2ff4b968`	`2ff4b968` (unchanged)	Deliberately older than gRPC 1.69.0's `b8b3e6e1`. Every later boringssl requires C11 `<stdalign.h>` on MSVC, which the Windows CI SDK (10.0.19041) doesn't ship.

Bazel 8

.bazelversion 7.5.0 → 8.8.1. Bzlmod stays off (--noenable_bzlmod --enable_workspace), so every external repo still comes from WORKSPACE and there is exactly one copy of protobuf, rules_java and rules_python.
--noincompatible_disallow_empty_glob: the root BUILD.bazel globs platform-specific paths that are empty on other platforms.
--noincompatible_disallow_ctx_resolve_tools: the 2021 rules_proto_grpc pin still calls ctx.resolve_tools.
glibc 2.17. Official Bazel 8 binaries need glibc ≥ 2.25 and can't start inside the manylinux2014 (CentOS 7) images that build Ray's Linux wheels. A new wanda image, ci/docker/bazel-glibc217, builds bazel_nojdk from the release's dist zip inside rayproject/manylinux2014 (devtoolset-10) and ships it with Temurin JDK 21. The ray-core, ray-cpp-core, ray-java and manylinux images put it behind bazelisk. Rebuilding Bazel only happens when .bazelversion changes. Wheels still target manylinux2014, so the user glibc floor is unchanged.
ci/ray_ci/test_bazel_sharding.py shares one mock workspace across its tests. Each test used to start a new Bazel server, which on Bazel 8 also re-fetches rules_python for autoloads and pushed it past its 60s timeout.

Declaration order: the first declaration of a repo wins

Several bumps only take effect because of where they are declared. In WORKSPACE, a repo keeps its first declaration: a maybe() skips a repo that already exists, and a later plain http_archive is ignored.

bazel_skylib / rules_cc are declared at the top of WORKSPACE. py_repositories() (rules_python 0.40.0) declares skylib 1.3.0 and rules_cc 0.0.13 before ray_deps_setup() runs, so the skylib pin in ray_deps_setup.bzl has had no effect since [build] Move rules_python off 0.9.0 #66207.
rules_java_dependencies() runs after ray_deps_setup(). It declares its own com_google_protobuf (29.0-rc2) through maybe(). If it ran first, it would silently replace Ray's pin.
rules_pkg is pinned to 0.9.1 before protobuf_deps() runs, which would otherwise declare 1.0.1. The v3.20.3 protobuf that rules_proto_grpc uses loads the root @rules_pkg//:providers.bzl stub, which was deleted in 0.10.0.
rules_proto 7.1.0 is declared before grpc_deps() for the same reason.

The versions that actually resolved were verified by reading each fetched repo's MODULE.bazel / version.json. bazel query does not catch these swaps.

Code changes needed by the new versions

.bazelrc defines PROTOBUF_FUTURE_REMOVE_CREATEMESSAGE, which compiles out protobuf's deprecated Arena::CreateMessage shim. GCC 10 (the manylinux2014 devtoolset) warns on the shim's own body in every file that includes arena.h, and -Werror makes that fatal.
JsonPrintOptions::always_print_primitive_fields → always_print_fields_with_no_presence (event.cc, one test). The export-event JSON does not change: unset proto3 optional fields were omitted before and still are.
cc_proto_library loads from @com_google_protobuf//bazel:cc_proto_library.bzl rather than @rules_cc (3 BUILD files).
gRPC's health service C++ target is health_cc_grpc in 1.69 (2 BUILD files).
UBSan: GCC rejects abseil's hash containers under the null, nonnull-attribute and returns-nonnull-attribute checks, so those three are off. GCC 11 also reports a false -Wmaybe-uninitialized in protobuf 29's generated move-assignment under ubsan; it is no longer fatal there.

gRPC 1.69 behaviour changes

Actor task ordering (real runtime bug, #67718). gRPC 1.69 leaves a channel IDLE for a moment after its first call starts. CoreWorkerClientPool evicts idle clients, so the client that sent an actor's first task could be evicted before the second task, which then went out on a new client with fresh sequence state. The actor ran the second task first and cancelled the first as stale (Task canceled due to stale sequence number). Fix: GrpcClient counts calls in flight and is only idle when none are. Local repro: 11/20 out of order before, 0/30 after.

gRPC INFO logs reach the driver (#67719). gRPC ≥ 1.65 logs through absl and leaves absl's INFO default alone when GRPC_VERBOSITY is unset, so lines such as chttp2_transport.cc ... Got goaway were forwarded to drivers as (raylet) .... Ray now sets absl's minimum level to ERROR unless GRPC_VERBOSITY is set.

Windows EventEngine. gRPC 1.69 turns on the EventEngine client and listener on Windows (1.58 marked them broken), and they abort at teardown (win_socket.cc "Check failed: is_shutdown_"). They stay off on Windows, and so does the EventEngine callback CQ: without it, callback-API code got no polling thread and pubsub_integration_test could never connect. Posix keeps 1.69's defaults.

Test timing. gRPC 1.69 makes some existing races easy to hit. test_broadcast (LLM) and the Serve replica-death test now let the dying replica's "accepted" message go out before it exits. The rest are in #67717.

Patches

Patch	Action
`protobuf-bazel7.patch`	Deleted. v29.6 has no `exec_tools` left.
gRPC's `third_party/protobuf.patch`	No longer applied. It only touches protobuf's Python package, which Ray doesn't build from Bazel.
`grpc-disable-layering-check.patch`	Ported (`gsm` → `csm`; fuzzers package gone). Without it macOS fails on gRPC's own missing deps.
`grpc-configurable-thread-count.patch`	Ported: `cpu.cc` moved to `src/core/util/linux/`. Behaviour unchanged.
`grpc-nextresult-cancelled-init.patch`	Ported.
`grpc-python-plugin-no-registered-method.patch`	New. gRPC's Python plugin (from 1.63) emits `_registered_method=True` and `server.add_registered_method_handlers()`, which need grpcio >= 1.64. The patch leaves both out, so the `*_pb2_grpc.py` stubs match what the 1.58 plugin generated and keep working with the `grpcio >= 1.42.0` that `setup.py` allows. Checked on grpcio 1.42.0, 1.62.3, 1.64.3 and 1.75.0; without the patch the stubs fail on 1.62 and 1.63.
`grpc-zlib-fdopen.patch`	Regenerated. The old hunk applied with fuzz to the wrong `http_archive`.
`grpc-cython-copts.patch`, `abseil-cpp-shadow.patch`	Unchanged.

What stays the same

Runtime protobuf floor for users. The Python and Java bindings are still generated by com_google_protobuf_rules_proto_grpc (v3.20.3).
grpcio floor for users. Still grpcio >= 1.42.0. See grpc-python-plugin-no-registered-method.patch above; CI pins grpcio 1.75, so only a test with an old grpcio shows this.
Wheel glibc floor. Still manylinux2014.
WORKSPACE. Bzlmod is a later step (required by Bazel 9).

Please review closely

gRPC 1.58 → 1.69 is 11 minor versions of core runtime. Watch Enable gRPC fork support in driver process #65984 / Use posix_spawn for specific-server to fix gRPC fork race #65982 (gRPC fork handling) and [core] Add tests for worker gRPC thread count selection #64898 (worker gRPC thread count). Release perf benchmarks on 1.69 are in the comments: all 13 core perf tests passed, with no regressions beyond what master nightly already shows.
macOS and Windows are verified by CI only.
bazel build //... on Bazel 8 reports conflicting actions for //cpp:plus.so / //cpp:counter.so (built in two test configurations). The targets CI builds are not affected. //release:requirements_* already fails to analyze on master's Bazel 7.5.

Tests run

Command	Result
Bazel 8.8.1: `bazel build --nobuild //:ray_pkg_zip //:ray_py_proto_zip //java:gen_ray_java_pkg //cpp:ray_cpp_pkg_zip`	analysis passes
Bazel 8.8.1: `bazel build //:ray_py_proto_zip //src/ray/gcs_rpc_client/tests:gcs_client_test`	build completed successfully (8,426 actions from cold)
Bazel 8.8.1: `bazel build --nobuild -k //...`	passes apart from the two known issues above
Premerge on Bazel 7.5 with every gate unblocked (sanitizers, aarch64, Windows, macOS)	green: build 75992
Release perf benchmarks (13 core tests)	passed: release build 113271
Premerge on Bazel 8.8.1 with every gate unblocked (sanitizers, aarch64, Windows, macOS)	green: build 76206 (733 jobs passed, no retries)

The Bazel 8 parts were first run in full CI on the PyO3 POC (#66875): all 14 native images built on Bazel 8 inside CentOS 7, and the remaining failures were the gRPC ones fixed here.

AI assistance was used for the analysis and for drafting this change.

🤖 Generated with Claude Code

gemini-code-assist

Code Review

This pull request upgrades Protobuf to v29.6, gRPC to v1.69.0, and Abseil to 20240722.0, along with updating rules_java, rules_cc, and bazel_skylib to ensure compatibility with Bazel 8. It also introduces tracking of in-flight RPC calls in GrpcClient to prevent channels from being prematurely marked as idle. Feedback on the changes highlights a critical race condition in GrpcClient where decrementing the in-flight call counter before executing the user callback could lead to premature client eviction or destruction while the callback is still running.

gemini-code-assist · 2026-09-29T18:03:55Z

+      num_calls_in_flight->fetch_sub(1);
+      user_callback(status, std::move(reply));


Decrementing num_calls_in_flight before invoking user_callback can allow the client to be detected as idle and evicted/destroyed by another thread (e.g., the client pool) while user_callback is still executing. If the callback accesses any resources associated with the client, this could lead to a use-after-free or race condition.

Invoking user_callback first and then decrementing the counter ensures that the client is kept alive and not considered idle until the callback has fully finished executing.

Suggested change

num_calls_in_flight->fetch_sub(1);

user_callback(status, std::move(reply));

user_callback(status, std::move(reply));

num_calls_in_flight->fetch_sub(1);

Agreed, fixed in 0c06a11: the callback now runs first and the in-flight count is decremented after it, so the client cannot look idle (and be evicted from CoreWorkerClientPool) while it is still handling the reply. //src/ray/rpc/tests:grpc_server_client_test (including TestNotIdleWhileCallInFlight) and //src/ray/core_worker_rpc_client/tests:core_worker_client_pool_test pass.

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

elliot-barn · 2026-11-02T22:58:32Z

Release perf benchmarks

All 13 core perf release tests (AWS) passed on 3ab4983a01: release build 113271.

Compared against the 2.59.0 baselines in release/perf_metrics/ and against master nightly runs since 10-12:

Area	vs 2.59.0
microbenchmark (mean of 5 runs)	Almost all metrics within −5% to +11%, with more gains than losses. Worst: `single_client_put_gigabytes` −4.8%. Best: `1_1_actor_calls_async` +10.6%
many_actors / many_pgs / many_tasks throughput	+7.8% / +5.3% / +48%
single_node, object_store, stress_test_dead_actors	within about ±7%, mostly faster
Dashboard p95/p99 latency (many_pgs, many_actors)	24–59% better

Two metrics are worse than 2.59.0, but master nightly already shows the same values without this PR:

Metric	2.59.0	This PR	Master nightly since 10-22
`stress_test_many_tasks` stage_0_time	5.9 s	10.4 s	8.4–11.7 s
`many_actors` dashboard p50 latency	9.2 ms	16.6 ms	14–26 ms

benchmark_worker_startup, callback_throughput, small_objects, large_objects and placement_group_performance_test have no checked-in baseline, so for those the only result is that they passed.

Premerge is fully green with every manual gate unblocked (sanitizers, aarch64, Windows, macOS): build 75924.

🤖 Generated with Claude Code

Ray's bazel_skylib 1.6.1 declaration in ray_deps_setup() never took effect: py_repositories() runs first and its maybe() declares bazel_skylib 1.3.0. Declare 1.7.1 at the top of WORKSPACE so it wins, and drop the dead declaration. Also pin rules_pkg to 0.9.1 ahead of protobuf_deps(). 0.9.1 is the newest release that still has the root providers.bzl stub the v3.20.3 protobuf used for Python/Java codegen loads. Both are versions the protobuf 29 / gRPC 1.69 bump (#66483) needs; landing them first keeps that PR smaller. rules_cc stays in #66483: 0.0.14 drops cc_proto_library and 0.0.15+ loads protobuf's bazel/ package, so it can only move together with protobuf. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

rules_java 7.x compiles Java at source/target 8 by default; 8.x defaults to 11. On the protobuf 29 / rules_java 8 branch (#66483) Ray's jars came out as class file version 55, which a Java 8 runtime rejects (UnsupportedClassVersionError in test_raydp and test_logging_java). Pin --java_language_version=8 now; under rules_java 7.x this matches the default and changes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

sai-miduthuri

Should this PR be stacked with 67717, 67718, 67719, 67720, 67721 ?

sai-miduthuri · 2026-11-05T19:22:14Z

-
-        os._exit(1)
+        import threading
+
+        # Exit shortly after this call returns, not inside it. The router only
+        # treats a request as delivered once the replica's "accepted" message
+        # arrives; os._exit() inside the call can kill the process before that
+        # message leaves it, so the router retries on the other replica, which
+        # then exits too and leaves no live replica.
+        threading.Timer(0.5, os._exit, args=(1,)).start()


Same comment as #67717 (comment)

Can this use asyncio.sleep + asyncio.create_task?

…lmod mode Not for merge. Proof of concept on top of #66483 and #66305, still on Bazel 7.5.0. - Turn on bzlmod alongside WORKSPACE (--enable_bzlmod --enable_workspace). - MODULE.bazel declares only rules_rust and rules_rust_pyo3 (0.74.0), so none of Ray's WORKSPACE repos change: com_google_protobuf stays 29.6 and the module graph's protobuf/rules_python live in separate canonical repos. - rust_probe/: a rust_library + rust_test, and a pyo3_extension consumed by a py_test as a data file. It cannot go in deps: the module world's rules_python PyInfo is a different provider than WORKSPACE's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

## Description Bumps two Bazel rule repos that the protobuf 29 / gRPC 1.69 bump (#66483) needs, so that PR is smaller. - **bazel_skylib 1.6.1 -> 1.7.1:** declared at the top of `WORKSPACE`. Ray's existing skylib 1.6.1 declaration in `ray_deps_setup()` never took effect: `py_repositories()` runs first, and its `maybe()` declares bazel_skylib 1.3.0. The first declaration of a repo wins, so it now has to come first. Protobuf 29 needs skylib >= 1.7.0 (`paths.is_normalized()`). - **rules_pkg 0.9.1:** pinned ahead of `protobuf_deps()`. It's the newest release that still has the root `providers.bzl` stub, which the v3.20.3 protobuf used for Python/Java codegen still loads. rules_cc stays in #66483. 0.0.14 drops `cc_proto_library`, and 0.0.15+ loads protobuf's `bazel/` package, which protobuf 23.4 doesn't have, so it can only move together with protobuf. ## Related issues Split out of #66483 (protobuf 29 / gRPC 1.69) to make that PR easier to review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…lmod mode Not for merge. Proof of concept on top of #66483 and #66305, still on Bazel 7.5.0. - Turn on bzlmod alongside WORKSPACE (--enable_bzlmod --enable_workspace). - MODULE.bazel declares only rules_rust and rules_rust_pyo3 (0.74.0), so none of Ray's WORKSPACE repos change: com_google_protobuf stays 29.6 and the module graph's protobuf/rules_python live in separate canonical repos. - rust_probe/: a rust_library + rust_test, and a pyo3_extension consumed by a py_test as a data file. It cannot go in deps: the module world's rules_python PyInfo is a different provider than WORKSPACE's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

## Description gRPC >= 1.65 logs through absl. When `GRPC_VERBOSITY` is unset, gRPC leaves absl's minimum log level alone, and absl defaults to INFO. gRPC INFO messages (e.g. "Got goaway") then reach stderr and get forwarded to the driver. Older gRPC defaulted to ERROR. `RayLog::StartRayLog` now sets absl's minimum log level to ERROR once per process, unless `GRPC_VERBOSITY` is set. Ray's own logging goes through spdlog and is unaffected. Note for reviewers: protobuf (23.4 today) also logs through absl, so landing this before the gRPC bump also hides protobuf's absl INFO and WARNING output. Master ends up in that state after the bump anyway. ## Related issues Split out of #66483 (protobuf 29 / gRPC 1.69) to make that PR easier to review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

## Description `GrpcClient::IsChannelIdleAfterRPCs()` treated a client as idle whenever its channel reported `IDLE` after the first call. gRPC moves a channel out of `IDLE` asynchronously once a call starts, so a client could report idle while a call was still in flight. The core worker client pool then evicts that client, and the next actor task goes out on a new client with fresh sequence state. The result is out-of-order or cancelled actor tasks. This PR: - Counts calls whose callback hasn't run yet, and only reports idle when that count is zero. The counter is a `shared_ptr`, so a reply that arrives after the client is destroyed can still decrement it. It is decremented only after the user callback runs, so the client can't look idle while it's still handling a reply. - Adds `TestNotIdleWhileCallInFlight`. - Makes `ReconnectionBackoff` time its no-reconnect window from when the channel enters `TRANSIENT_FAILURE`, instead of after a fixed 1s wait. The 2s backoff has up to 20% jitter, so the old window could overlap the first retry. On gRPC 1.58 the race is rare. On 1.69 the channel stays `IDLE` noticeably longer after the first call, and actor-ordering tests failed often. ## Related issues Split out of #66483 (protobuf 29 / gRPC 1.69) to make that PR easier to review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…lmod mode Not for merge. Proof of concept on top of #66483 and #66305, still on Bazel 7.5.0. - Turn on bzlmod alongside WORKSPACE (--enable_bzlmod --enable_workspace). - MODULE.bazel declares only rules_rust and rules_rust_pyo3 (0.74.0), so none of Ray's WORKSPACE repos change: com_google_protobuf stays 29.6 and the module graph's protobuf/rules_python live in separate canonical repos. - rust_probe/: a rust_library + rust_test, and a pyo3_extension consumed by a py_test as a data file. It cannot go in deps: the module world's rules_python PyInfo is a different provider than WORKSPACE's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

## Description Fixes four tests that depend on timing. They pass today but failed consistently, on some platforms, once gRPC was bumped to 1.69 in #66483. Each change makes the test wait for the state it assumes, instead of hoping that state is already reached: - `test_release_resource`: the raylet can grant leases out of submission order, so the canary task could take the CPU before `f`. Wait until `f` is running before submitting the canary. (On Windows the canary won every time.) - `test_broadcast` (llm): `self_destruct` called `os._exit()` inside the call, which can kill the replica before its "accepted" message reaches the router. The router then retries on the other replica, which also exits. Exit shortly after the call returns instead. - `test_capacity_queue_router`: same pattern. Let the crashing replica's acceptance through before it exits. - `TestEvictLineageStopsAtTarget`: pass `max_retries` inline. The lambda used a local `const int` without capturing it, which GCC and Clang allow for a constant but MSVC rejects. All test-only. No runtime changes. ## Related issues Split out of #66483 (protobuf 29 / gRPC 1.69) to make that PR easier to review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: elliot-barn <sim@local> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

rules_java 8.x (needed for Bazel 8) loads @com_google_protobuf//bazel:*.bzl, which does not exist in the v23.4-era protobuf Ray pinned, so the two move together. gRPC < 1.69 builds a standalone @upb that does not compile against protobuf 29, so gRPC (and abseil, boringssl to match it) moves too. - Declare bazel_skylib 1.7.1, rules_cc 0.0.16 and rules_java 8.14.0 at the top of WORKSPACE. The first declaration of a repo wins, and rules_python's py_repositories() otherwise pulls skylib 1.3.0 and rules_cc 0.0.13. - Run rules_java_dependencies() after ray_deps_setup(), so its maybe() does not replace Ray's protobuf with 29.0-rc2. - Pin rules_pkg 0.9.1: the v3.20.3 protobuf used by rules_proto_grpc loads the root @rules_pkg//:providers.bzl stub, which rules_pkg 0.10.0 removed. - Arena::CreateMessage -> Arena::Create, always_print_primitive_fields -> always_print_fields_with_no_presence (same output for Ray's protos). - Load cc_proto_library from protobuf; use gRPC's health_cc_grpc target. - Drop protobuf-bazel7.patch, grpc-disable-layering-check.patch and grpc's protobuf.patch; port the thread-count, NextResult and zlib-fdopen patches. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

- rules_java 8.14.0 -> 8.12.0: from 8.13.0 on, the prebuilt java_tools need GLIBCXX_3.4.21, newer than manylinux2014's libstdc++ (jar build). - Keep boringssl at 2ff4b968: later commits require C11 <stdalign.h> on MSVC, which the Windows CI SDK (10.0.19041) lacks (Windows build). - Restore grpc-disable-layering-check.patch, ported to 1.69: gRPC's own BUILD files miss deps that clang enforces on macOS (macOS build). - Define PROTOBUF_FUTURE_REMOVE_CREATEMESSAGE: GCC 10 warns on the deprecated shim's own body in every file including arena.h, and -Werror makes that fatal (all manylinux wheel / cpp / java images). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

rules_java 7.x compiled Java at source/target 8 by default; 8.x defaults to 11. Ray's jars came out as class file version 55, which a Java 8 runtime rejects (UnsupportedClassVersionError in test_raydp and test_logging_java). Pin --java_language_version=8. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…240722.2 - Keep gRPC's EventEngine client and listener off on Windows. gRPC 1.69 enables them there (1.58 marked them broken), and they abort at teardown: win_socket.cc "Check failed: is_shutdown_" (test_implicit_resource, test_object_spilling_2_asan). Posix keeps 1.69's defaults. - abseil 20240722.0 -> 20240722.2: patch 1 fixed a security issue in hash container create/resize. - UBSan: GCC rejects abseil's hash containers under the null, nonnull-attribute and returns-nonnull-attribute checks (a function-pointer comparison in a constexpr stops being a constant expression). Turn those three off; they must follow -fsanitize=undefined in the same flag list. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

The Windows patch moved gRPC's client and listener back to iomgr but left event_engine_callback_cq on. On Windows grpc_iomgr_run_in_background() returns IsEventEngineCallbackCqEnabled(), so gRPC assumed I/O makes progress in the background and gave callback-API code no polling thread, while the iomgr sockets need one. pubsub_integration_test, which only uses the callback API, could never connect ('tcp handshaker shutdown' after 20s). Raylet/GCS only worked because their CompletionQueue threads poll the same iomgr. Turn the callback CQ off too so callback-API code uses CallbackAlternativeCQ, which polls iomgr, as 1.58 did. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

GCC 11 reports a false -Wmaybe-uninitialized in protobuf 29's generated move-assignment (CanMoveWithInternalSwap) when it is inlined into core_worker.cc under the ubsan config, and -Werror turns it into a build failure for core_worker_lib. Keep the warning but don't make it fatal under ubsan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Bazel's own Linux releases need glibc >= 2.25 since Bazel 8, so they cannot start in manylinux2014 (CentOS 7, glibc 2.17), where Ray builds its Linux wheels. Add a bazel-glibc217 wanda image that builds the .bazelversion release from source inside that image (bootstrap with compile.sh, then //src:bazel with an embedded JDK) and smoke-tests a server start there. The core, cpp-core and java image builds and the manylinux CI image copy it in and point bazelisk at it. Wheels keep targeting glibc 2.17. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

On x86_64 the second-stage //src:bazel build runs rules_java's prebuilt ijar, which needs GLIBCXX_3.4.21 and fails on CentOS 7. Ship compile.sh's bazel_nojdk with the Temurin JDK 21 it was built with, passed via --server_javabase. Embed the release label so 'bazel info release' (and anything keyed on the version) reports , not no_version, which is what failed the aarch64 smoke test. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Each test built a fresh mock workspace and HOME, so every bazel query started a new Bazel server, and on Bazel 8 fetched rules_python from the registry to autoload the mock BUILD's py_test. Six of those took the small test past its 60s timeout (69s locally). Nothing writes to the workspace, so make the fixture module-scoped: 21.5s locally. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Move from Bazel 7.5.0 to 8.8.1, still in WORKSPACE mode (bzlmod stays off, so every repo still comes from WORKSPACE and there is one copy of protobuf, rules_java and rules_python). - .bazelversion 7.5.0 -> 8.8.1. - --noincompatible_disallow_empty_glob: the root BUILD.bazel globs platform-specific paths that are empty on other platforms. - --noincompatible_disallow_ctx_resolve_tools: the 2021 rules_proto_grpc pin still calls ctx.resolve_tools. - Pin rules_proto 7.1.0 ahead of grpc_deps(): 6.0.2 reads native proto_common.ProtoLangToolchainInfo, which Bazel 8 removed. Official Bazel 8 binaries need glibc >= 2.25 and can't start in the manylinux2014 images, so the previous commits build Bazel for glibc 2.17 in CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Bazel 8 autoloads sh_binary/sh_test from rules_shell, and on Windows they need a registered shell toolchain to build their launcher. Bzlmod registers it automatically; in WORKSPACE mode nothing did, so every Windows job failed analysis on @bazel_tools//tools/test:collect_coverage ("No suitable shell toolchain found"). Declare rules_shell 0.2.0 (protobuf 29's pin) early and call rules_shell_toolchains(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…lmod mode Not for merge. Proof of concept on top of #66483 and #66305, still on Bazel 7.5.0. - Turn on bzlmod alongside WORKSPACE (--enable_bzlmod --enable_workspace). - MODULE.bazel declares only rules_rust and rules_rust_pyo3 (0.74.0), so none of Ray's WORKSPACE repos change: com_google_protobuf stays 29.6 and the module graph's protobuf/rules_python live in separate canonical repos. - rust_probe/: a rust_library + rust_test, and a pyo3_extension consumed by a py_test as a data file. It cannot go in deps: the module world's rules_python PyInfo is a different provider than WORKSPACE's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

The *_pb2_grpc.py stubs come from the Python plugin of the gRPC that Ray builds. From 1.63 on, that plugin emits `_registered_method=True` on client stubs and `server.add_registered_method_handlers()`, which only exist in grpcio >= 1.64. With gRPC 1.69 the stubs failed on grpcio 1.62 (client and server) and 1.63 (server), while setup.py allows grpcio >= 1.42.0. CI pins grpcio 1.75, so it never showed. Patch the plugin to leave both out, so the stubs match what the 1.58 plugin generated. This gives up grpcio's registered-method fast path, which Ray's stubs did not use before either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…lmod mode Not for merge. Proof of concept on top of #66483 and #66305, still on Bazel 7.5.0. - Turn on bzlmod alongside WORKSPACE (--enable_bzlmod --enable_workspace). - MODULE.bazel declares only rules_rust and rules_rust_pyo3 (0.74.0), so none of Ray's WORKSPACE repos change: com_google_protobuf stays 29.6 and the module graph's protobuf/rules_python live in separate canonical repos. - rust_probe/: a rust_library + rust_test, and a pyo3_extension consumed by a py_test as a data file. It cannot go in deps: the module world's rules_python PyInfo is a different provider than WORKSPACE's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 99c1471. Configure here.}

cursor · 2026-11-12T23:08:16Z

+          out->Print("metadata)\n");
+         }
+       }
+     }


Plugin patch misses version gate

High Severity

The new plugin patch only strips _registered_method=True and add_registered_method_handlers(). gRPC 1.69's Python plugin still emits a GRPC_GENERATED_VERSION check that rejects grpcio below 1.69.0 at import, so *_pb2_grpc.py stubs fail on the grpcio >= 1.42.0 floor setup.py still allows. CI will not catch this because the lockfile pins grpcio 1.78.0.

Additional Locations (1)

bazel/ray_deps_setup.bzl#L382-L388

^{Reviewed by Cursor Bugbot for commit 99c1471. Configure here.}

elliot-barn added the go add ONLY when ready to merge, run all tests label Sep 25, 2026

elliot-barn changed the title ~~[build] Bump protobuf to v29.6, rules_java to 8.14.0 and gRPC to 1.69.0~~ [build] Bump protobuf to v29.6, rules_java to 8.12.0 and gRPC to 1.69.0 Sep 25, 2026

elliot-barn marked this pull request as ready for review September 29, 2026 18:02

elliot-barn requested review from a team, MengjinYan and edoakes as code owners September 29, 2026 18:02

gemini-code-assist Bot reviewed Sep 29, 2026

View reviewed changes

cursor Bot reviewed Sep 29, 2026

View reviewed changes

Comment thread src/ray/util/event.cc Resolved

ray-gardener Bot added the core Issues that should be addressed in Ray Core label Sep 29, 2026

elliot-barn force-pushed the elliot-barn/bazel-protobuf-29 branch from d96817d to b82fc29 Compare October 8, 2026 18:20

cursor Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread thirdparty/patches/grpc-windows-disable-event-engine.patch Resolved

This was referenced Oct 8, 2026

[POC][build] Build a PyO3 extension with rules_rust_pyo3 in hybrid bzlmod mode #66870

Closed

[POC][build] Hybrid bzlmod on Bazel 8.8.1 with rules_rust_pyo3 (stacks on #66483) #66875

Draft

elliot-barn requested a review from a team as a code owner October 23, 2026 17:59

elliot-barn requested a review from a team as a code owner October 23, 2026 20:33

elliot-barn mentioned this pull request Oct 28, 2026

[DO NOT MERGE] diag: Windows lease order for test_release_resource #67496

Closed

elliot-barn force-pushed the elliot-barn/bazel-protobuf-29 branch from 3ab4983 to bb53ec3 Compare November 2, 2026 23:25

elliot-barn changed the title ~~[build] Bump protobuf to v29.6, rules_java to 8.12.0 and gRPC to 1.69.0~~ [build] Bazel 8.8.1: bump protobuf to v29.6, rules_java to 8.12.0 and gRPC to 1.69.0 Nov 4, 2026

ELundby45 mentioned this pull request Nov 4, 2026

ray rebuild for win-arm64 (bazel 8) AnacondaRecipes/ray-packages-feedstock#30

Merged

sai-miduthuri reviewed Nov 5, 2026

View reviewed changes

elliot-barn force-pushed the elliot-barn/bazel-protobuf-29 branch from 8444a22 to cef9fdd Compare November 8, 2026 21:03

elliot-barn mentioned this pull request Nov 12, 2026

[ci] Explain the glibc 2.17 Bazel build and mark it for removal #68113

Draft

elliot-barn and others added 11 commits November 12, 2026 22:43

elliot-barn force-pushed the elliot-barn/bazel-protobuf-29 branch from c867d2a to 59c6539 Compare November 12, 2026 22:44

cursor Bot reviewed Nov 12, 2026

View reviewed changes

Navigation Menu

Repository navigation

[build] Bazel 8.8.1: bump protobuf to v29.6, rules_java to 8.12.0 and gRPC to 1.69.0 - #66483

[build] Bazel 8.8.1: bump protobuf to v29.6, rules_java to 8.12.0 and gRPC to 1.69.0#66483
elliot-barn wants to merge 12 commits into
masterfrom
elliot-barn/bazel-protobuf-29

elliot-barn commented Sep 25, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Sep 29, 2026

Uh oh!

elliot-barn Oct 20, 2026

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

elliot-barn commented Nov 2, 2026

Uh oh!

sai-miduthuri left a comment

Uh oh!

sai-miduthuri Nov 5, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Nov 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		num_calls_in_flight->fetch_sub(1);
		user_callback(status, std::move(reply));

Conversation

elliot-barn commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Split for review

Versions

Bazel 8

Declaration order: the first declaration of a repo wins

Code changes needed by the new versions

gRPC 1.69 behaviour changes

Patches

What stays the same

Please review closely

Tests run

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

elliot-barn Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elliot-barn commented Nov 2, 2026

Release perf benchmarks

Uh oh!

sai-miduthuri left a comment

Choose a reason for hiding this comment

Uh oh!

sai-miduthuri Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Nov 12, 2026

Choose a reason for hiding this comment

Plugin patch misses version gate

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

elliot-barn commented Sep 25, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading