Fix installation and build on current Ubuntu and arm64 by Toby1009 · Pull Request #264 · apache/incubator-resilientdb

Toby1009 · 2026-07-22T18:34:54Z

Closes #245

The problem

The README states OS Requirements: Ubuntu 20+ and tells users to run
./INSTALL.sh. I could not get that to work on Ubuntu 24.04 on either
architecture, nor on 26.04/arm64, and on arm64 it fails before Bazel is
installed at all.

Reproduction, on a machine that has sudo, curl and wget, which is what a
normal Ubuntu install has:

docker run --rm -it --platform linux/arm64 ubuntu:24.04 bash
apt-get update && apt-get install -y sudo curl wget gnupg ca-certificates git
git clone https://github.com/apache/incubator-resilientdb && cd incubator-resilientdb
./INSTALL.sh; echo "exit=$?"

On master this prints a series of failures and then:

exit=0

Several commands failed, but the script has no set -e and kept going. The
final apt-get install python3-dev succeeded, so the run exits 0 and reports
success. Bazel is not installed. The failure surfaces later as a confusing
error from a build that cannot start.

What actually breaks

1. Bazel cannot be installed on arm64. INSTALL.sh adds the Bazel apt
repository with [arch=amd64]. That repository's Release file declares
Architectures: amd64 and has no arm64 index, so apt selects bazel:amd64 and
then cannot satisfy g++:amd64:

 bazel:amd64 : Depends: g++:amd64 but it is not installable
E: Unable to correct problems, you have held broken packages.

The script then falls back to compiling Bazel 6.0.0 from source, and that fails
too, for a different reason on each release. Bazel 6.0.0 vendors dependencies
about as old as this repository's own, so it hits the same problems:

On 24.04 (GCC 13), while compiling Bazel's bundled Abseil:

external/com_google_absl/absl/strings/internal/str_format/extension.h:34:33:
  error: found ':' in nested-name-specifier, expected '::'

On 26.04 (GCC 15), while compiling Bazel's bundled zlib:

external/net_zlib_zlib/gzwrite.c:89:20: error: implicit declaration of
  function 'write'; did you mean 'fwrite'?

Two smaller things in the same run: apt-key is deprecated on 22.04 and 24.04
and no longer exists on 26.04, where it reports apt-key: command not found
(the keyring line above it already does the same job); and python3.10-dev is
hardcoded, so on 24.04 the run also reports E: Unable to locate package python3.10-dev.

2. Abseil 20211102.0 does not compile with GCC 13+. It declares fixed-width
integer types without including <cstdint>. This is the same failure the
bundled copy inside Bazel produces above.

3. 53 headers in the core tree have the same problem. They name uint64_t
and friends and rely on another header to provide <cstdint>. (ecosystem/ is
a separate Bazel workspace excluded via .bazelignore; its one affected header
is covered in the GraphQL issue below.)

4. The vendored zlib misses <unistd.h>. ./configure normally sets
Z_HAVE_UNISTD_H in zconf.h; Bazel compiles the sources directly, so the
gz* sources call write()/close()/lseek() undeclared. GCC 14 turned that
into an error.

5. kv_service_tools spins forever on arm64. getopt_long returns int
and signals the end of the options with -1, but the result was stored in a
char. GCC on Linux/AArch64 defaults plain char to unsigned, so -1 became
255, the loop condition never became false, and the tool sat at 100% CPU
inside getopt_long while the replicas reported socket recv:0:

#0  getopt_long (argc=11, ..., options="h", ...) at ./posix/getopt1.c:33
#1  main ()

This affects every option-style invocation, which is the interface the README
documents. The older positional get/set/getvalues/getrange path returns
before this loop and is unaffected. GCC defaults char to signed on x86-64,
which is why it was never seen there.

Why these fixes and not the obvious ones

zlib is not upgraded. The obvious fix is a version bump, and it does not
work. Compiling gzwrite.c with GCC 15:

zlib	no define	`-DZ_HAVE_UNISTD_H`
1.2.12	fails	builds
1.3.2	fails	builds

The guard in zconf.h depends on the build configuration rather than on the
release, so the pin is left alone and the define is added instead.

Abseil is not moved to the newest release. 20250127.0 and later select on
@rules_cc//cc/compiler, which the rules_cc bundled with Bazel 6.0.0 does not
provide; the build fails during analysis. 20240722.2 built successfully in the
matrix below. The tree's only direct Abseil labels are absl/status and
absl/status:statusor (common/BUILD:21), and the bump needed no ResilientDB
source changes.

Bazelisk is not a new idea. Docker/Dockerfile_mac:39 already installs
Bazel this way for arm64, though unpinned. It reads .bazelversion, so the
pinned Bazel 6.0.0 is unchanged; only how it is obtained differs. Here it is
pinned to v1.29.0 and verified against the SHA-256 published with that release
before being installed.

All 53 headers get <cstdint>, not only the ones that break today. Every
changed header directly names a fixed-width integer type. Some fail now and
others currently compile through a transitive include; adding the direct
include makes that explicit rather than dependent on what other headers happen
to pull in. Taking each header's own #include <...> lines and compiling them
against a uint64_t declaration, 43 of the 53 cannot obtain the type that way
and 10 can.

Verification

Each row was run from a clean container: ./INSTALL.sh, then
bazel build //service/kv:kv_service.

OS	arch	GCC	Python	install	build
22.04	arm64	11.4.0	3.10	ok	ok, 404 actions
24.04	amd64	13.3.0	3.12	ok	ok, 406 actions
24.04	arm64	13.3.0	3.12	ok	ok, 404 actions
26.04	arm64	15.2.0	3.14	ok	ok, 636 actions

22.04 is the oldest release tested here; the README also lists 20.04, which I
did not test. Every run reports the Bazel 6.0.0 pinned in .bazelversion.

The path the README describes also completes on 26.04/arm64, which it did not
before:

./INSTALL.sh                                                # exit 0, bazel 6.0.0
./service/tools/kv/server_tools/generate_keys_and_certs.sh  # exit 0, 17 files
./service/tools/kv/server_tools/start_kv_service.sh         # exit 0, 5 nodes
bazel build service/tools/kv/api_tools/kv_service_tools     # exit 0

bazel-bin/service/tools/kv/api_tools/kv_service_tools \
  --config service/tools/config/interface/service.config \
  --cmd set_with_version --key key1 --value hello_apache --version 0
    set key = key1, value = hello_apache, version = 0 done, ret = 0
    current value = value: "hello_apache"
    version: 1

bazel-bin/service/tools/kv/api_tools/kv_service_tools \
  --config service/tools/config/interface/service.config \
  --cmd get_with_version --key key1 --version 0
    get key = key1, value = value: "hello_apache"
    version: 1

shellcheck -s sh (0.10.0), checkbashisms and dash -n are clean on
INSTALL.sh. clang-format --dry-run --Werror reports no header that was
clean on master and is not clean here.

Two additions to the apt list deserve a word: ca-certificates is required for
the curl download of Bazelisk, and default-jre-headless plus zip/unzip
are what dev/check-license needs to run Apache RAT — Bazel itself ships its
own JDK and does not need them.

Why CI did not catch any of this

build.yml:45 and ut.yml:45 pin Python 3.10, and build.yml:62 and
ut.yml:59 pin CC=gcc-11, which is what Ubuntu 22.04 ships — so the GCC
13/14 failures cannot appear in the PR-gating workflows. The one arm64 build
that does exist is build-push.yml's Docker image job: it runs post-merge on
master only, is skipped when Docker Hub credentials are absent, uses an Ubuntu
20.04 base (old toolchain, so the compiler failures cannot appear there
either), and builds kv_service_tools without ever running it, so the getopt
spin stays invisible too.

No workflow is changed or added here. Coverage that uses the distribution's own
toolchain would stop this recurring, but that is a decision about CI policy and
runner budget rather than part of this fix, and it is easier to discuss on its
own. Happy to send it separately.

A note on architecture support

Bazelisk publishes only linux-amd64 and linux-arm64, so the rewritten
script stops with a clear message on any other Debian architecture rather than
failing part way through. The previous source-compilation fallback was not
architecture-limited in principle, although it does not complete on any release
I tested. If ppc64el or s390x matter, that needs a different way of obtaining
Bazel and I have not attempted it.

Deliberately not included

bazel build ... still fails, because pybind11 2.6.2 does not compile
against Python 3.11+. //service/kv:kv_service has no pybind11 in its
dependency graph and is unaffected. Filed as [BUG] pybind11 2.6.2 does not build against Python 3.11+, so bazel build ... fails on current Ubuntu #262.
ecosystem/graphql/WORKSPACE:224 fetches this project with
git_repository(branch = "master"), so a GraphQL build uses remote master
rather than the checkout and none of these fixes reach it. Changing how a
subproject depends on its parent is an architectural decision for the
maintainers. Filed as ecosystem/graphql builds against remote master rather than the checkout it lives in #263.
Docker/Dockerfile, INSTALL/README.md and the Ansible role each
install Bazel their own way, one of them still using apt-key. Consolidating
them is a separate change.
rapidjson-dev and protobuf-compiler appear to have no consumer left
in the core tree, but they predate this change and removing them is unrelated
cleanup.

Commits

The dependency pinning in commit 5 is independent of the rest: protobuf moves
off a mutable git tag, four archives that had no sha256 get one, and
bazel_skylib is pinned to the 1.0.3 that already wins today so that the
choice is deliberate rather than a side effect of macro ordering. Drop that
commit if it is out of scope for this PR; nothing else depends on it.

These headers declare uint64_t, uint32_t and friends without including <cstdint>, relying on some other header to pull it in transitively. libstdc++ stopped providing it through unrelated headers, so from GCC 13 on the build fails with "'uint64_t' does not name a type". The rule applied here is include-what-you-use: every header in the core tree that names a fixed-width integer type includes <cstdint> itself, whether or not it happens to compile today, so that compiling does not depend on what other headers happen to pull in. ecosystem/ is a separate Bazel workspace excluded via .bazelignore and is not touched here; its one affected header is noted in a separate issue.

Abseil 20211102.0 declares fixed-width integer types without including <cstdint>, so it stops compiling from GCC 13 on: absl/strings/internal/str_format/extension.h:34:33: error: enum class FormatConversionChar : uint8_t; found ':' in nested-name-specifier, expected '::' 20240722.2 builds with the Bazel pinned in .bazelversion. 20250127.0 and later select on @rules_cc//cc/compiler, which the rules_cc bundled with Bazel 6.0.0 does not provide, and fail during analysis; releases between the two were not tested. The tree's only direct Abseil labels are absl/status and absl/status:statusor, and the bump needed no ResilientDB source changes. The first zlib URL now 404s because zlib.net serves only the current release from the top level and moves older ones under fossils/. The build survives on the bazel-mirror fallback but logs a download failure on every fetch.

zlib's ./configure normally rewrites zconf.h to set this. Bazel compiles the sources directly, so zconf.h skips <unistd.h> and the gz* sources call write(), close() and lseek() undeclared: external/net_zlib_zlib/gzwrite.c:89:20: error: implicit declaration of function 'write'; did you mean 'fwrite'? GCC 14 turned implicit function declarations into errors, and the -w already in this file does not suppress them. Upgrading zlib does not help: 1.2.12 and 1.3.2 both fail without the define and both build with it, because the guard in zconf.h depends on the build configuration rather than on the release. The version pin is therefore left alone.

getopt_long returns int and signals the end of the options with -1, but the result was stored in a char. GCC on Linux/AArch64 defaults plain char to unsigned, so -1 became 255, the loop condition never became false, and the tool spun at 100% CPU inside getopt_long without ever reaching the network code: #0 getopt_long (argc=11, ..., options="h", ...) at ./posix/getopt1.c:33 #1 main () Every option-style invocation hung on arm64 as a result, which is the interface the README documents, while the replicas reported "socket recv:0". The older positional get/set path returns before this loop is reached and is unaffected. GCC defaults char to signed on x86-64, which is why this was never seen there. The only arm64 build in CI is the post-merge Docker image job, which compiles this tool on an Ubuntu 20.04 base and never runs it, so the spin was invisible there too. The three sibling tools in this repository already declare this variable as int.

protobuf was fetched with git_repository(tag = "v3.10.0"). A tag can be moved to a different commit upstream, which would silently change what is built and is not detectable from the workspace. It is now an http_archive of the release tarball with a sha256, which also removes the need for a git binary during the fetch. hedron_compile_commands, duckdb, pybind11_bazel and rules_boost had no sha256 at all, so their contents were trusted on every fetch. duckdb even carried a comment asking for the hash to be filled in. bazel_skylib was never declared by this project. Abseil needs config_setting_group from lib/selects.bzl, which the skylib that protobuf 3.10 pins does not provide, so the build only worked because rules_foreign_cc_dependencies() happens to register 1.0.3 before protobuf_deps() runs. 1.0.3 is pinned explicitly here: it is the version that already wins today, so resolution is unchanged, only deliberate.

On a current Ubuntu several steps failed but the script kept going, and the final apt-get succeeded so the whole run still exited 0; the failure only surfaced later as a confusing build error. It now sets -eu and stops at the first problem. Bazel was installed from the Bazel apt repository, whose Release file declares "Architectures: amd64" and has no arm64 index. On arm64 apt selected bazel:amd64 and then could not satisfy g++:amd64. The fallback path compiled Bazel from source, which fails on GCC 14+ because Bazel 6.0.0 vendors a zlib that calls write()/close() undeclared. Bazelisk replaces both: it is pinned to v1.29.0 and verified against the SHA-256 published with that release, then reads .bazelversion and fetches the same Bazel 6.0.0 as before, so the pinned version is unchanged. This is the approach Docker/Dockerfile_mac already uses. Other fixes: - apt-key is deprecated on 22.04 and 24.04 and no longer exists on 26.04. The keyring line above it already does the same job, so the call is gone. - The script is run as root in containers and by cloud-init, where sudo is usually absent. sudo is now used only when not already root. - build-essential was installed only on the fallback path, so a normal run never got a C++ compiler. - python3.10-dev was hardcoded; python3-dev already follows the system. - hooks/pre-push does not exist in this repository, so the unconditional rm and ln -s left a dangling symlink. The link is now made only if the hook exists. - The shebang was below the licence header and therefore inert, while ecosystem/deployment/ansible runs this script with sh. It is now #!/bin/sh on line 1 and the script is POSIX (shellcheck -s sh and checkbashisms are clean). - Bazelisk is downloaded as the invoking user into a temporary directory and verified before sudo install writes it, so an interrupted transfer cannot leave a truncated binary in /usr/local/bin. - The script cds to its own directory so Bazelisk reads this checkout's .bazelversion rather than whatever is in the caller's working directory. - A trapped signal whose handler does not exit lets the script continue: the old trap cleaned up on INT/TERM and then carried on. Signals now exit. Note on architecture support: Bazelisk publishes only linux-amd64 and linux-arm64, so the script now stops with a clear message on anything else. The previous source-compilation fallback was not architecture-limited in principle, though it does not complete on any tested release.

Toby1009 · 2026-07-24T06:02:04Z

For release planning context: #245, which this PR closes, is on the Release v1.13.0 Planning milestone, but this PR is not attached to it.

@msadoghi since you added #245 to that milestone, would you like this tagged for v1.13.0 as well? I don't have permission to set it myself.

Happy to split the PR up if the six commits are easier to review separately, or to rebase if anything has moved underneath it.

Toby1009 · 2026-07-24T16:55:17Z

Thanks for helping trigger the CI!
My local testing didn’t cover the full set of targets exercised by CI, but I’ve identified the issue and found a fix. I’ll update the PR shortly.

msadoghi · 2026-07-24T17:02:01Z

For release planning context: #245, which this PR closes, is on the Release v1.13.0 Planning milestone, but this PR is not attached to it.

@msadoghi since you added #245 to that milestone, would you like this tagged for v1.13.0 as well? I don't have permission to set it myself.

Happy to split the PR up if the six commits are easier to review separately, or to rebase if anything has moved underneath it.

Added to our next milestone as requested.

The Abseil bump in 4efc7cd (20211102.0 -> 20240722.2) deletes absl/base/internal/thread_annotations.h, the header that defined the unprefixed thread-safety annotations (GUARDED_BY on line 64 of the 20211102.0 release). block_manager.h used GUARDED_BY while including no thread-annotations header of its own -- its only Abseil include was absl/status/status.h -- so the macro reached it transitively via cord.h -> cordz_info.h -> absl/base/thread_annotations.h. In 20240722.2 that last header no longer defines the unprefixed spelling, so the annotation is parsed as a declaration: ./platform/consensus/ordering/poc/pow/block_manager.h:96:52: error: ISO C++ forbids declaration of 'GUARDED_BY' with no type The current spelling is ABSL_GUARDED_BY, from absl/base/thread_annotations.h; both expand to __attribute__((guarded_by(x))). That header still arrives transitively, so the rename alone would compile, but it is now included directly and the target takes the @com_google_absl//absl/base:core_headers dependency that owns it, so the annotation no longer rides on Abseil's internal include graph. This was the only live use in the core tree; the three remaining occurrences of these macros sit inside comments. //service/kv:kv_service does not depend on poc/pow, so building that target alone -- as this branch's earlier checks did -- never reached the failure; both PR-gating workflows build `...`, where it broke. With this commit `bazel build ...` completes on ubuntu-24.04 with gcc-11 11.5.0 and Python 3.10.20 on amd64, and in the same environment on arm64.

Toby1009 · 2026-07-25T01:01:42Z

Pushed e3a181c0, which fixes the bazel build ... failure both build workflows hit.

What broke: the Abseil bump in this PR (20211102.0 -> 20240722.2) deletes absl/base/internal/thread_annotations.h, the header that defined the unprefixed GUARDED_BY. platform/consensus/ordering/poc/pow/block_manager.h used it while including no thread-annotations header of its own, so the macro had been arriving transitively via absl/status/status.h -> cord.h -> cordz_info.h -> absl/base/thread_annotations.h. The fix switches to ABSL_GUARDED_BY, includes that header directly, and adds the matching @com_google_absl//absl/base:core_headers dep. Both macros expand to __attribute__((guarded_by(x))), so this is a spelling change only.

I missed it locally because my earlier checks built //service/kv:kv_service, and

bazel query "somepath(//service/kv:kv_service, //platform/consensus/ordering/poc/pow:block_manager)"

is empty — that target never reaches the PoW code. Both PR-gating workflows build ..., which does.

Verified before pushing this time:

environment	result
`bazel-build CI`, GitHub Actions, ubuntu-24.04 amd64, runner image 20260720.247.2	`Build completed successfully, 1803 total actions` (run)
`Unite Test`, GitHub Actions, same runner image	`Build completed successfully, 1803 total actions` (run)
Ubuntu 24.04 container, amd64, gcc-11 11.5.0, Python 3.10.20, Bazel 6.0.0	1803 / 1803 actions
same container, arm64	1798 / 1798 actions

The two Actions runs are this repo's unmodified build.yml and ut.yml, triggered from a branch in my fork on the same runner image version as the failing run here. The failing build stopped at action 1655 of 1803; all 1803 now complete, so nothing else was hiding behind the first error.

On the check license failure: it is Security.md and nothing else, and it fails identically on unmodified master:

$ git checkout 7c714dff        # master
$ docker run --rm -v "$PWD":/github/workspace -w /github/workspace \
    ghcr.io/apache/skywalking-eyes/license-eye:latest header check
INFO  Totally checked 1384 files, valid: 1251, invalid: 1, ignored: 132, fixed: 0
ERROR the following files don't have a valid license header:
Security.md

Same counts as the run on this PR. It is pre-existing and unrelated — this PR does not touch Security.md. Happy to send a separate one-line PR adding the ASF header to it if that would be useful.

Toby1009 added 6 commits July 23, 2026 02:30

Toby1009 mentioned this pull request Jul 24, 2026

[BUG] Incompatibility and Build Failures with Docker Ubuntu:latest [ 24.04 ] and ARM64 Architecture #245

Open

msadoghi assigned msadoghi and unassigned msadoghi Jul 24, 2026

msadoghi added this to the Release v1.13.0 Planning milestone Jul 24, 2026

Toby1009 mentioned this pull request Jul 25, 2026

[DO NOT MERGE] CI verification run for PR #264 Toby1009/incubator-resilientdb#1

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix installation and build on current Ubuntu and arm64 - #264

Fix installation and build on current Ubuntu and arm64#264
Toby1009 wants to merge 7 commits into
apache:masterfrom
Toby1009:fix/install-sh-arm64

Toby1009 commented Jul 22, 2026

Uh oh!

Toby1009 commented Jul 24, 2026

Uh oh!

Toby1009 commented Jul 24, 2026

Uh oh!

msadoghi commented Jul 24, 2026

Uh oh!

Toby1009 commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Toby1009 commented Jul 22, 2026

The problem

What actually breaks

Why these fixes and not the obvious ones

Verification

Why CI did not catch any of this

A note on architecture support

Deliberately not included

Commits

Uh oh!

Toby1009 commented Jul 24, 2026

Uh oh!

Toby1009 commented Jul 24, 2026

Uh oh!

msadoghi commented Jul 24, 2026

Uh oh!

Toby1009 commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants