[RFC] Exploring an end-to-end ARM64 support path for AgentENV: findings from a validated DGX Spark deployment

[leether]

Affected component

Cross-cutting / other

RFC intent and possible closure

This is a request for design feedback, not a request to declare ARM64 supported
or to merge one large implementation. A useful outcome could be any of the
following:

1. ARM64 server/runtime support is currently out of scope; the issue can close
   with that position recorded.
2. The project is open to an initial support tier; the issue can close after
   that tier and its compatibility boundary are recorded and focused follow-up
   issues or PRs are identified.
3. The project needs more evidence before choosing a direction; the issue can
   close or be deferred with the missing validation and revisit conditions
   recorded.

Problem statement

This RFC asks whether ARM64 server/runtime support is within AgentENV's scope
and, if so, what initial level of support the project might intend to provide
and test.

A server that compiles on aarch64 is an important milestone, but it does not
by itself prove that the selected Firecracker build, guest kernel, tools drive,
ublk/OverlayBD path, seccomp policy, and snapshot lifecycle work together.

There are two adjacent open contributions as of 2026-07-31:

• #86 proposes ARM64 release
  artifacts, installer selection, and multi-architecture container publishing.
• #94 proposes an ARM64
  source-build path, architecture-aware Firecracker and guest-kernel downloads,
  ARM tools-drive construction, and openEuler recognition.

This issue is not a replacement for either PR. It first asks whether the
project would like to pursue ARM64 server/runtime support. If the answer is
yes, this case study may help distinguish a build or installation milestone
from an end-to-end supported runtime.

We independently assembled and operated one complete AgentENV stack on an
NVIDIA DGX Spark. This work did not use or validate #86 or #94. It demonstrates
that one full ARM64 combination is feasible, but it does not prove that every
ARM64 host, kernel, or distribution is compatible.

Related work and scope

Item	What it covers	What remains outside its stated validation
#60	General request for ARM support	The intended support boundary and acceptance criteria
#63	Template build failure caused by using an incompatible upstream Firecracker binary	A complete compatible dependency set and snapshot regression
#86	Prebuilt ARM server release, installer selection, multi-arch container	End-to-end runtime behavior is outside its stated scope
#94	ARM source build, architecture-aware dependencies, tools drive, openEuler	Prebuilt ARM server and multi-arch publishing are excluded; native ARM unit and integration tests were not run
This issue	One independent native ARM runtime case and a possible staged path	It does not request adoption of one downstream implementation

One dependency observation may be relevant when reviewing those paths. At the
current #94 head (3134f27), its manifest selects
firecracker-1.15.1-patch-v1-aarch64 and vmlinux-6.1.175-aarch64 from the
aenv-deps
release. That release tag resolves to source commit
90288c3,
whose serial restore path predates
#5764 and does
not save and restore the full UART device state.

This is a source/provenance comparison, not a claim that I executed #94's
published binary or reproduced a failure through #94. It is evidence that an
ARM snapshot baseline should be selected and tested explicitly instead of
being inferred from an asset name.

Use case

The concrete case is a single-node AgentENV server on a native ARM64 host. If
the project later chooses to pursue an end-to-end ARM runtime tier, a supported
profile would need to:

1. install or build every host and guest runtime artifact for the same
   architecture;
2. resolve an OCI image through the declared OverlayBD/ublk storage tier;
3. build a template and restore sandboxes from its snapshot;
4. sustain repeated envd RPCs and console output after restore;
5. pause and resume a sandbox;
6. recover envd after a crash or SIGKILL; and
7. reject or deliberately migrate an incompatible snapshot before starting
   Firecracker.

The independently validated profile was:

Layer	Validated value
Host hardware	NVIDIA DGX Spark, bare metal, 20 ARM CPUs (Cortex-X925 / Cortex-A725), 4 KiB pages
Host OS	Ubuntu 24.04, Linux 6.17.0-1021-nvidia, aarch64
Host capabilities	/dev/kvm present; CONFIG_BLK_DEV_UBLK=m; real uvm-ublk-daemon and OverlayBD data path
AgentENV assembly	0.1.0-arm64.11
AgentENV validated production source	d79fd2d
Public reference branch	leether/arm64-assembly-line
Firecracker base	kvcache-ai/firecracker-next v1.17.0-next.1 / fee2681
Firecracker exact build	b792de1, aarch64-unknown-linux-gnu
Firecracker runtime ID	fc-v1.17.0-next.1+agentenv.arm64.1.b792de1
Firecracker release	agentenv-v1.17.0-next.1-arm64.1
Guest kernel	Ubuntu/NVIDIA 6.17.0-1021-nvidia ARM64 kernel image
Tools drive	downstream 0.1.0-arm64.3
envd	0.6.13 / bfdbb24

This is one validated combination, not a proposal that AgentENV adopt these
exact versions.

Current behavior and workarounds

End-to-end runtime evidence

The public reference branch contains a native ARM regression that deliberately:

• keeps UART TX active while taking a snapshot;
• checks that envd stderr is not /dev/console;
• writes 16 KiB directly to /dev/console after restore, so log redirection
  cannot hide a broken virtual UART;
• completes 100 consecutive envd RPCs after restore; and
• kills envd and verifies supervisor recovery without recreating the sandbox.

The automated gate is documented in
arm64-assembly.md
and implemented in
uart_snapshot.rs.

Separate production-promotion observations on this single DGX host recorded
three snapshot-based starts at 90 ms, 79 ms, and 71 ms; 300/300 repeated execs;
two 16 KiB console-write checks; pause/resume; and envd supervisor recovery.
These are operator-recorded observations, not additional iterations encoded in
the public automated test.

Preliminary single-host performance context

End-to-end CLI pilot

We also ran a small warm-cache pilot on the same DGX host, using the same 2
vCPU / 1024 MiB sandbox size. Snapshot starts used the default alpine
template; cold starts used localhost:5000/alpine:3.20 from the local registry.
The modes were interleaved, and no vLLM requests were running during the
measurement.

End-to-end CLI measurement	Snapshot (n=5)	Cold start (n=5)
Successful create, first exec, and delete	5/5	5/5
start -d p50	105.7 ms	541.7 ms
start -d range	101.0–287.5 ms	531.8–714.5 ms
Start to first successful exec p50	179.9 ms	613.0 ms

Within this one host and measurement boundary, snapshot start p50 was about
5.1x faster than cold start, and time to first successful exec was about 3.4x
faster. Both modes had a slower first iteration, so this sample is too small to
claim p95 or p99. A publishable benchmark should first run five warm-up
iterations per mode and then measure at least 100 iterations.

This pilot reports only two end-to-end measurements: start -d, and start to
first successful exec. Those values include CLI, API, orchestration, and
readiness overhead and are not used below to judge the README latency targets.

Repository snapshot benchmark

AgentENV's current
README
states that snapshot-backed environments boot or resume in under 50 ms, pause
in under 100 ms, and create incremental snapshots in under 100 ms even under
heavy disk modification. The same repository now contains a direct backend
snapshot_benchmark.rs
with explicit operation boundaries. The README does not say that this file
produced the published numbers, so this is a reproduction of the repository's
closest code-defined benchmark, not a claim about the maintainers' original
benchmark environment.

We copied that benchmark file unchanged at commit a33198e and verified the
source SHA256 (abc15b944e54639774868a43a4ebf46c52b8fe62309878de4f98418565d6aa6b)
before building it against the ARM reference branch. The timed operations,
1-vCPU setting, base 128 MiB guest, and heavy-data preparation were unchanged.
Firecracker, the guest kernel, tools drive, and envd were the same artifacts as
the validated production assembly. The benchmark used an isolated AgentENV
home, runtime directory, ublk daemon, image cache, network CIDRs, and
ubuntu:24.04 rootfs. Other production services remained running.

We then ran the repository's full Criterion mode unchanged: 3 seconds of
warm-up, a 20-second target measurement window, and 10 samples per benchmark.
Criterion extended several windows when necessary. Its output reports
[lower bound, point estimate, upper bound]:

Code-defined backend measurement	README target	Criterion result	Observation
Hot snapshot_resume	<50 ms	[78.530, 82.124, 85.371] ms	Target not reproduced
snapshot_creation (sandbox.pause())	<100 ms	[47.602, 52.598, 58.493] ms	Under target
snapshot_creation_1gdisk (1 GiB direct write, sync, drop caches)	<100 ms under heavy disk modification	[47.376, 51.445, 55.950] ms	Under target
Cold snapshot_resume	No separate README threshold	[103.61, 107.66, 111.47] ms	Additional context
snapshot_creation_1gmem	README claim is about disk modification	[272.64, 302.17, 333.07] ms	Additional context
50-way concurrent_resume	No separate README threshold	[317.68, 324.41, 331.88] ms	Additional context

To check repeatability outside Criterion's aggregation, we launched three new
benchmark processes and retained all 30 bounded samples for both cold and hot
resume. Each round produced exactly one roughly 30-second stall: cold sample 6
and hot sample 7. Across all samples, cold resume had median 106.2 ms and hot
resume median 80.9 ms, but each had a 3/30 stall rate. The 27 sub-second cold
samples were 91.70–118.71 ms (mean 103.51 ms); the 27 sub-second hot samples
were 61.55–85.06 ms (mean 76.95 ms). The retained stalls were
30.91–31.04 seconds cold and 30.52–30.54 seconds hot. Their duration is close
to the benchmark's explicit 30-second socket timeout, but the current logs do
not establish which wait consumed it; that is correlation, not a root-cause
claim. Criterion's summarized interval did not expose this periodic stall, so
both the central estimate and the bounded raw-sample behavior are reported.

The benchmark times FirecrackerSandbox::resume_from_snapshot_config() and
sandbox.pause() directly. It excludes CLI/API/orchestrator latency and does
not separately time user-facing persistent snapshot publication or object
storage. The file also has no separate snapshot-boot case; hot and cold resume
are the closest code-defined paths. Within that exact backend boundary, the
base and heavy-disk capture paths met the two numerical under-100-ms thresholds,
while hot resume did not meet the under-50-ms threshold. This is code-path
evidence, not a full reproduction of every README lifecycle boundary. It is
also not an ARM-versus-x86 comparison; that requires the same commit, artifacts,
runner, and background-load policy on a comparable x86 host.

Compatibility lessons from the bring-up

1. AgentENV needs its patched Firecracker API. A same-numbered upstream
   binary may run but reject AgentENV's downstream drive fields such as
   direct, as Failed to build template on Arm64 server. #63 demonstrates.

2. Snapshot behavior has a minimum baseline. Our earlier
   v1.15.1-patch-v1-lineage assembly reproduced a deterministic UART TX stall
   after snapshot restore. The behavior matches the bug fixed by Firecracker
   #5764 and disappeared after moving to a v1.17.0-next.1 lineage containing
   that fix. We did not perform a single-commit bisect, so this should be read
   as a strong compatibility finding, not proof that only one commit differed.

3. A guest workaround can mask a device bug. Keeping envd logs away from
   /dev/console prevents the original HTTP 504 symptom, but it does not prove
   that UART TX survived restore. The regression therefore exercises the
   console device directly.

4. The build target and seccomp policy are coupled. The validated
   Firecracker is an ARM GNU build. Reusing the ARM musl policy unchanged did
   not cover libc-specific signal and process behavior; an empty fallback would
   silently weaken isolation. The release therefore carries a reviewed,
   default-deny ARM GNU policy and fails if it is missing.

5. The guest kernel is part of the product. Our selected distribution
   kernel has CONFIG_IP_PNP disabled, so the tools init applies the final
   AgentENV-injected ip= argument in userspace. The 6.1 ARM reference config
   in kvcache-ai/firecracker enables CONFIG_IP_PNP. Either approach can be
   made to work, but the required capability or fallback needs to be explicit.

6. Snapshot identity is larger than a version string. A downstream runtime
   ID is useful provenance, but it is not by itself a complete compatibility
   key. ARM restore compatibility can also depend on CPU features, GIC version,
   host/KVM behavior, Firecracker snapshot format, guest kernel, and guest tools.
   Restoring multiple clones also duplicates guest userspace state. Firecracker's
   VMGenID helps the kernel RNG only when the ARM guest has DeviceTree VMGenID
   support (upstream Linux added it in 6.10; a 6.1 guest needs a backport), and
   it does not generically rotate userspace tokens, machine identity, or cached
   random state.

Desired behavior

If the project chooses to pursue ARM64 server/runtime support, a possible
outcome would be a documented set of support tiers with observable acceptance
criteria:

• Build tier: the server and tools drive compile for ARM64, and selected
  artifacts are architecture-correct and checksummed.
• Install tier: a declared host profile can install or provision the
  complete dependency set with preflight checks and a reproducible BOM.
• Runtime tier: native ARM KVM tests cover template creation,
  OverlayBD/ublk attachment, active-UART snapshot restore, repeated envd
  execution, pause/resume, and supervisor recovery.
• Release tier: the project publishes supported profiles, exact dependency
  provenance, security policy, upgrade/rebuild rules, rollback guidance, and
  downloadable test evidence.

Disabling ublk may be a valid reduced profile, as #94 proposes for openEuler
kernels without CONFIG_BLK_DEV_UBLK, but the documentation should identify
which snapshot, on-demand-loading, and OverlayBD guarantees are then
unavailable. A build-only profile and an end-to-end runtime profile should not
share the same unqualified support label.

Proposed approach

Would maintainers be open to discussing whether ARM64 server/runtime support
belongs in AgentENV's roadmap? If so, the following questions may help define a
staged path:

1. Project scope: Is ARM64 server/runtime support something the project
   would like to pursue? If so, should the initial goal be build-only, source
   installation, prebuilt releases, or an end-to-end supported runtime? Which
   tier would feat: add ARM64 support for AgentENV #86 or feat: add ARM (aarch64) platform support #94 satisfy if merged?
2. Reference profiles and preflight: Which host distributions, kernel/KVM
   versions, CPU/GIC combinations, page sizes, cgroup mode, and ublk capability
   should be supported or checked before host setup?
3. Versioned dependency contract: What are the canonical source and release
   coordinates for Firecracker, guest kernel, tools/envd, ublk daemon, and
   OverlayBD? Who would own downstream patches, upstream synchronization,
   provenance, and end-of-life?
4. Guest lifecycle contract: Which kernel options should be required, when
   should a userspace ip= fallback be supported, and how should ARM snapshot
   clones handle VMGenID, entropy, boot_id, machine identity, and
   application-level unique state before user code runs?
5. Snapshot compatibility: Which hardware and software fields should form
   the compatibility key? How should placement avoid incompatible hosts, and
   should legacy or mismatched snapshots fail closed, warn, or use an explicitly
   tested compatibility map?
6. Build and isolation targets: Should the canonical ARM64 Firecracker
   target be GNU, musl, or both? Should production startup fail closed when a
   matching default-deny seccomp policy is absent? What checksum, signing, and
   SBOM model should release artifacts use?
7. Native validation: If the project pursues a runtime or release tier,
   where could KVM + ublk ARM tests run, and which results should block release?
   Suggested coverage includes clean install, real OverlayBD/ublk I/O, template
   build, active-UART restore, repeated envd RPCs, pause/resume,
   clone-uniqueness checks, snapshot mismatch, missing seccomp policy, and a
   clearly defined latency benchmark.

If maintainers consider ARM64 support in scope and indicate an initial tier,
the public reference can be split into focused contributions. We can provide
the tested case, help validate #86/#94 on DGX Spark, and implement or review
agreed pieces; this issue does not ask for one large downstream merge.

Compatibility and operational impact

API/config changes:

• Explicit support tiers may require capability reporting and preflight output.
• A snapshot compatibility record may require new metadata fields beyond one
  Firecracker runtime ID.

Snapshot or storage format changes:

• Legacy snapshots need an explicit reject, migrate, or compatibility-map
  policy.
• Clone-uniqueness handling may require a post-restore guest lifecycle hook.

New host/runtime requirements:

• Native ARM64 KVM for a runtime tier.
• ublk/OverlayBD only for tiers that promise those storage features.
• A guest kernel and seccomp policy matching the declared profile.

Upgrade and rollback considerations:

• Changing Firecracker snapshot format, guest kernel, or tools drive may require
  template rebuilds.
• Canary promotion and rollback are useful reference operational practices,
  but need not be ARM-specific acceptance criteria for an initial source-build
  tier.

Alternatives considered

1. Comment only on feat: add ARM (aarch64) platform support #94. That would mix an independent runtime case with
   review of one source-build implementation and could be misread as validation
   of that PR.
2. Comment only on feat: add ARM64 support for AgentENV #86. The independent assembly did not use feat: add ARM64 support for AgentENV #86, and feat: add ARM64 support for AgentENV #86's
   release scope does not define runtime compatibility.
3. Report only the UART incident. Firecracker already fixed the device bug;
   the remaining AgentENV question is dependency selection and regression
   ownership.
4. Publish only the downstream recipe. One working BOM does not define
   portable support tiers, snapshot placement, clone uniqueness, or lifecycle
   policy.

Non-goals

• This issue does not claim that feat: add ARM64 support for AgentENV #86 or feat: add ARM (aarch64) platform support #94 has been validated on DGX Spark.
• It does not assume that AgentENV has already committed to supporting ARM64.
• It does not claim that one DGX Spark validates all ARM CPUs, GIC versions,
  kernels, page sizes, or distributions.
• It does not present the five-run CLI pilot, one Criterion run, or 30 bounded
  resume samples as a cross-architecture or production tail-latency benchmark.
• It does not assume that the repository benchmark is the undisclosed original
  methodology behind the README numbers; it reports the exact code boundary
  and commit used for comparison.
• It does not require AgentENV to adopt the downstream kernel, exact version
  set, release scripts, or runtime-ID implementation unchanged.
• It does not reopen the Firecracker UART bug fixed in #5764.
• It does not make canary promotion and rollback prerequisites for accepting an
  ARM source-build contribution.

Contribution

I can help test or review it.

Pre-submission checklist

• I searched existing issues and discussions. When will ARM platform be supported? #60, Failed to build template on Arm64 server. #63, feat: add ARM64 support for AgentENV #86, and feat: add ARM (aarch64) platform support #94 are
  adjacent work, but none currently defines the possible end-to-end ARM64
  runtime support path discussed here.
• I described a concrete problem and use case, not only a proposed
  technology.
• I understand that acceptance of the problem does not imply approval of a
  particular implementation.

[yingdi-shan]

Thank you, this analysis and the arm64 benchmark data are very valuable!