shen-zig

shen-zig is a Shen 41.2 implementation written for Zig 0.16.0. It provides a complete bytecode runtime, a native AOT backend, exact reference counting with cycle recovery, checked allocation reuse, and parallel independent isolates.

The project is currently version 0.1.0. The implementation is complete for the scope described below, but the public API and generated AOT ABI should still be treated as experimental.

Status

Implemented:

• Shen OS Kernel 41.2 loading and execution
• all required KLambda primitives
• lexical Core IR and explicit control-flow register IR
• ownership insertion with verified clone/move decisions
• bytecode compiler and VM
• readable Zig AOT generation
• exact non-atomic RC and immediate acyclic reclamation
• exact stop-the-world cycle collection
• Cons, vector, string, and closure reset/reuse
• checkpoint arenas and Prolog rollback workspaces
• independent Runtime/Isolate heaps running on separate threads
• identity-preserving cyclic graph transfer between isolates
• RC auditing, allocator-failure tests, and bounded-memory soak tests

Intentionally not included:

• JIT compilation
• a shared mutable heap between threads
• concurrent garbage collection
• moving or compacting collection

Parallel execution uses independent isolates with non-atomic RC. Values cross isolate boundaries by graph copying or immutable message snapshots.

Requirements

• Zig 0.16.0
• a platform supported by the selected Zig target

The Shen 41.2 KLambda sources required by the build are included under vendor/ShenOSKernel-41.2.

Quick start

Build and evaluate an expression:

zig build -Doptimize=ReleaseFast
zig build run -Doptimize=ReleaseFast -- --eval "(+ 20 22)"

Start a REPL:

zig build run -Doptimize=ReleaseFast -- --repl

Build and run the AOT-enabled executable:

zig build aot -Doptimize=ReleaseFast
zig build aot-run -Doptimize=ReleaseFast -- --eval "(+ 20 22)"

The ordinary executable is installed as zig-out/bin/shen. The AOT build generates native Zig functions for the kernel and links them into zig-out/bin/shen-aot.

Command-line interface

shen --check-kernel
shen --boot
shen --eval EXPRESSION
shen --repl
shen --soak [ITERATIONS]

Optional flags:

• --audit-rc recomputes incoming reference counts from the complete root and object graph.
• --no-reuse disables the reuse pass and runtime reuse operations without changing program semantics.

An optional positional kernel directory may replace the vendored default. Shen code may also invoke (collect-cycles) explicitly.

Architecture

Shen source
    -> upstream Shen compiler
KLambda
    -> lexical Core AST
    -> virtual-register control-flow graph
    -> forwarding / reuse / ownership / verification
       |-> stackification -> owned bytecode -> VM
       `-> readable Zig source -> native AOT functions
    -> exact RC heap + arenas + exact cycle sweep

klambda/pipeline.zig is the backend-independent compilation pipeline. Ownership is decided on virtual registers before bytecode stack slots or AOT locals are selected. A later verifier rejects unclassified uses, use-after- move, invalid handler flow, and unsafe reuse candidates.

The backend-neutral execution/ layer owns callable identity, native ABI frames, currying, and the bounded tail-call dispatcher. Native and bytecode functions share stable function IDs and can call each other. Dynamic definitions remain bytecode and supersede an installed AOT definition with the same name.

The VM keeps its opcode, call, return, and unwind transitions in one auditable state machine. Dynamic generation lifetime, native bridging, frame storage, stable dynamic names, and root scratch storage are separate components.

Memory and ownership

Value is a tagged u64. Small integers, booleans, nil, and special values are immediate. Managed values use one non-moving object header and explicit ownership transfers.

Runtime ownership follows five operations:

alloc
retain
release
replace
collectCycles

Function arguments are normally consumed and return values transfer ownership. release uses an iterative work queue, so long lists do not recurse through the native stack. Object child edges are declared once in runtime/object.zig; RC, cycle collection, isolate copying, and message reconstruction all consume that topology.

Object descriptors use fixed-slot slab pages. Variable string, vector, closure, symbol, and bigint payloads use size-class or page-rounded storage. Pages carry occupancy information so the complete heap can be enumerated without moving objects.

Cycle collection marks from Runtime and active VM roots, rebuilds counts for survivors, and directly destroys unreachable objects without releasing dead edges recursively. Automatic scans run at an allocation threshold, when the VM becomes idle, and before one allocation retry after OOM.

Reuse is an optimization only. It requires a unique candidate and compatible layout or capacity. --no-reuse removes it while retaining the ordinary RC correctness model.

AOT

The AOT backend consumes the same verified ownership CFG as the bytecode compiler. Generated Zig keeps ownership transfers, reuse sites, handler blocks, root safepoints, and tail calls visible.

The full kernel is split into 16 compilation shards by default:

zig build aot -Doptimize=ReleaseFast -Daot-shards=16

Sharding bounds compiler memory and incremental rebuild scope; it does not create multiple runtimes. A serialized, bounds-checked bytecode image is also embedded so dynamic bytecode islands can execute without unbounded native/bytecode recursion.

Isolates

Each Isolate owns one Runtime and one managed heap. Only its bound thread may enter it. A stopped isolate can be rebound to another thread, while multiple independent isolates may execute concurrently.

Cross-isolate graph copying and immutable Message snapshots preserve object identity, sharing, closures, and cycles. Host streams are rejected as NonTransferableResource instead of being shared implicitly.

Verification

Run the unit suite:

zig build test

Useful targeted checks:

zig build run -Doptimize=ReleaseFast -- --check-kernel
zig build aot-check -Doptimize=ReleaseFast
zig build soak -Doptimize=ReleaseFast
zig build official -Doptimize=ReleaseFast
zig build aot-official -Doptimize=ReleaseFast

The complete acceptance target runs unit, bounded-memory, VM official, AOT build, and AOT official checks:

zig build acceptance -Doptimize=ReleaseFast

The official 134-test Shen suite has completed in VM and AOT modes with reuse both enabled and disabled. Tests also cover cyclic graphs, randomized reachability comparisons, exact RC recounting, exception cleanup, dynamic code retirement, allocator failure at Runtime and VM allocation points, isolate migration, parallel heaps, and long-session memory bounds.

Build targets

Target	Purpose
run	Build and run the bytecode VM
test	Run unit and architecture tests
aot-check	Compile a representative generated AOT slice
aot	Generate and compile the full native kernel
aot-run	Run the AOT-enabled executable
soak	Run bounded-memory REPL/typecheck/Prolog workloads
official	Run all 134 tests through the VM with and without reuse
aot-official	Run all 134 tests through AOT with and without reuse
acceptance	Run all release acceptance gates

Source layout

src/
  runtime/      Value, heap, RC, roots, arenas, cycles, isolates
  klambda/      reader, Core/register IR, ownership and reuse passes
  execution/    backend-neutral callables, ABI, dispatcher
  vm/           bytecode, compiler, verifier, execution state machine
  aot/          function analysis and Zig source generation
  primitives/   KLambda primitive registry and implementations

License

The shen-zig port code is available under the Zero-Clause BSD License.

The vendored Shen OS kernel is Copyright (c) 2010-2022 Mark Tarver and remains under its own BSD 3-Clause license. Source and binary distributions must preserve the notices required by that license. The name of Mark Tarver may not be used to endorse or promote derived products without prior written permission.