Muxiva

A Rust-native, real-time multimodal agent runtime with one graph and lifecycle contract across Rust, C++, Python, and TypeScript.

简体中文 · Documentation · Architecture · Flagship voice demo · Developer manual · Agent integration · Studio · Observability · Graph v1 · Release operations · Testing

Muxiva is an early-stage runtime for building streaming voice, video, text, and binary agents as static processing graphs. Rust owns scheduling, bounded queues, backpressure, lifecycle, cancellation, signals, events, shutdown, and observability. Nodes and adapters can be implemented in Rust, C++, Python, or TypeScript without moving language-specific objects across runtime boundaries.

The project currently provides a tested Runtime foundation and an application-layer Qwen + Agora real-voice flagship. It is not yet a production-ready agent platform.

Why Muxiva

• One runtime core: scheduling and safety semantics live in Rust.
• One data model: immutable Frame values carry audio, video, text, bytes, signals, and events.
• Bounded by design: queues, media duration, bytes, in-flight work, and shutdown deadlines have explicit limits.
• Language isolation: C ABI handles, Python execution domains, and Node.js workers prevent foreign code from running on RTC or Rust scheduling threads.
• Deterministic lifecycle: prepare, process, finish, abort, cancellation, and late-result behavior are explicit and tested.
• One graph protocol: programmatic builders, JSON Graph v1, the CLI, and the local Studio share the same graph definition.

Project status

Muxiva is pre-alpha. Stages 1–11 of the foundation plan are implemented, but several public APIs and integrations remain intentionally limited.

Area	Status	Current boundary
Frames, graph model, sync/concurrent runtime	Available	Static DAGs; exact port and frame types
Backpressure and real-time flow control	Available	Bounded queues, audio merge, managed streams
Signal and NotificationBus control	Available	Explicit adjacent Signal routing; process-local observable Events
C ABI and C++ SDK	Available	Versioned ABI, RAII wrappers, installable CMake package, and hosted Graph v1 text factories
RTC Nodes	Experimental	Shared-session Agora C++ audio/data ingress and egress; live credential certification remains
Media normalization	Experimental	Optional FFmpeg streaming audio resampling plus RGBA8/I420 scale and color conversion
Python/PyO3 package	Experimental	Dedicated thread/asyncio loop and hosted Graph v1 text factories; isolated_process is rejected
Node-API package	Experimental	Dedicated Worker and hosted Graph v1 text factories; Promise-returning transforms are rejected
JSON Graph v1 and CLI	Experimental	Exact-version Registry compilation, concurrent execution of compiled-in factories, bounded waits, initialization, and local Studio
Local Studio	Available	Node Lab, typed wiring, Python Host, C++ ABI packs, project experiences, local Run/Stop
Model Nodes	Experimental	Qwen Python Node Packs are vendor adapters outside Core

Architecture

Read the diagram from top to bottom: product surfaces declare a Graph and discover Node Factories; the vendor-neutral Rust Core compiles and executes it; Rust, C++, Python, and TypeScript Nodes provide replaceable capabilities; RTC, model APIs, and token services remain outside Core. Solid lines are data or calls, dashed magenta lines are Signal control, and dotted gray lines are process-local NotificationBus telemetry.

The runtime never treats ASR, LLM, TTS, transport, codec behavior, or a vendor “provider” as Core responsibilities. See the system overview and core-concepts walkthrough, or open the editable Draw.io source.

Quick start

Install the muxiva CLI once

After the first public release, macOS ARM64 and Intel users install the tested native binary without Rust:

brew install PiyotaHu/muxiva/muxiva
muxiva --version

Until that release is published, use the source installation:

git clone https://github.com/PiyotaHu/muxiva.git muxiva
cd muxiva
cargo install --locked --path crates/muxiva-cli
muxiva --version

Source installation requires Rust stable as selected by rust-toolchain.toml. CMake, CPython/maturin, and Node.js/pnpm are contributor prerequisites only for their corresponding SDKs. After either one-time install, normal usage never needs cargo run -p ... or knowledge of the Rust workspace.

Run the real voice assistant

The flagship application offers Qwen Audio Realtime plus Demo 2, an inspectable full-duplex Qwen Server VAD + ASR → independently versioned Pi TypeScript coding Agent → Speech Formatter → cancellable Qwen TTS graph, with Agora C++ transport, a browser microphone, and real workspace-scoped file tools:

./examples/voice-agent/setup.sh       # macOS: downloads and verifies Agora SDK
cp examples/voice-agent/.env.example examples/voice-agent/.env
# macOS local development: opens Studio by default
./examples/voice-agent/run.sh
# Linux, Docker, or a production-style local test:
./examples/voice-agent/run.sh --headless
# With headless mode running, start the independent browser client:
cd examples/voice-agent && npm run voice-room

Open http://127.0.0.1:4173, test the default Backend URL http://127.0.0.1:8080, then start the conversation. The Linux Runtime, HTTP Bootstrap API, standalone web deployment, two-identity RTC model, and remote/Docker paths are documented in the headless deployment guide and flagship voice demo guide.

Create, validate, and run a graph

muxiva init my-agent
muxiva validate my-agent
muxiva run my-agent
muxiva serve my-realtime-agent

muxiva init creates a complete project directory. muxiva validate is side-effect free: it never creates or starts a Node. muxiva run compiles the graph against the built-in Registry, materializes every exact Factory selection, and executes it through the concurrent Runtime. Runs have a 30-second default deadline; use --timeout-ms and --shutdown-timeout-ms to set bounded execution and cleanup waits. muxiva serve is the long-running, headless contract for real-time Graphs. It exposes only a health endpoint and browser RTC bootstrap endpoint; Studio is not required.

Start the local visual Studio

muxiva studio

With no argument, Studio discovers the current project; from the Muxiva source root it opens the flagship Voice Agent. Studio opens a bundled visual Graph v1 editor. Drag Nodes from the Palette, wire compatible typed ports, open ◎ Observe to identify slow Nodes and backed-up Edges, or open Create Node to edit and register a project Node without leaving the browser. Text Python project Nodes run through the trusted local development Host. Studio listens on 127.0.0.1 by default and generates a local access token. See the Studio guide and observability guide.

Build and test the language SDKs

./scripts/check-python.sh
./scripts/check-node.sh
./scripts/check-ffi.sh

These scripts build real installable packages, run integration tests, and execute independent Python, TypeScript, and C++ consumer examples. See the developer manual for Agent integration and language-specific Node workflows.

Flagship graphs

The real-voice Realtime and Cascade templates live under examples/voice-agent/.muxiva/templates/. run.sh defaults to Studio on macOS/Windows shells and Headless Runtime on Linux. Use --studio or --headless to select explicitly. The checked-in examples/voice-agent/graph.json is shared by both modes.

Graph JSON is declarative configuration. It cannot contain executable code, dynamic scripts, credentials, or arbitrary remote resources. See the Graph and typed ports guide.

Repository layout

muxiva/
├── crates/
│   ├── muxiva-types/       # Immutable frames, IDs, values, errors
│   ├── muxiva-core/        # Graph, runtime, queues, flow and control plane
│   ├── muxiva-ffi/         # Versioned C ABI
│   ├── muxiva-graph-json/  # Graph v1 parser and compiler
│   ├── muxiva-cli/         # muxiva command-line interface
│   ├── muxiva-studio/      # Local token-authenticated Studio server
│   ├── muxiva-python/      # PyO3/maturin package
│   ├── muxiva-node/        # Node-API native module
│   └── muxiva-testkit/     # Deterministic test harnesses
├── bindings/node/        # @muxiva/core package
├── bindings/agent/       # Vendor-neutral @muxiva/agent TypeScript contract
├── cpp/                  # Public C/C++ SDK
├── providers/            # Vendor integrations: Qwen/Python and Agora/C++
├── examples/             # Rust, graph, Python, TypeScript and C++ examples
├── fuzz/                 # Fuzz targets
├── docs/                 # Design, testing and pre-release reports
└── scripts/              # Reproducible quality gates

Quality gates

The commands below are for Muxiva contributors working on the repository, not for application developers using the installed muxiva binary.

Run the consolidated local gate:

./scripts/check-quality.sh

Individual gates include:

./scripts/check-rust.sh
./scripts/check-ffi.sh
./scripts/check-ffi-asan.sh
./scripts/check-rtc.sh
./scripts/check-rtc-asan.sh
./scripts/check-python.sh
./scripts/check-node.sh
./scripts/check-cpp-consumer.sh
./scripts/check-bench.sh

The test framework covers deterministic graph faults, queue pressure, managed-stream cancellation, foreign execution domains, ABI ownership, Mock RTC shutdown, CLI/Studio authorization, and port conflicts. Optional Miri, fuzz, and TSan scripts report an explicit SKIP when the required toolchain is unavailable.

Roadmap

Near-term priorities:

1. Stabilize public Rust, C++, Python, and TypeScript SDK contracts.
2. Stabilize the new schema-driven multimodal Source, Transform, Sink, and named multi-output foreign Factory APIs.
3. Stabilize Studio observability thresholds and certify Prometheus/OTLP compatibility with hosted backends.
4. Run and retain D09 Agora live-room certification on each release platform; extend D08 into compressed codec/device providers.
5. Implement versioned Python process isolation and TypeScript Promise support.
6. Publish packages, compatibility matrices, performance baselines, and release artifacts.

Real provider integrations should remain adapters or nodes and must not become mandatory Core dependencies.

Contributing

Design feedback, bug reports, reproducible test cases, and focused pull requests are welcome. Read CONTRIBUTING.md, the Code of Conduct, and the governance model before participating. Public API, Graph or Manifest Schema, Runtime, Studio, CLI, provider, and architecture changes must update docs/ in the same pull request.

Please keep changes bounded, deterministic, and free of real service credentials. New foreign-language, RTC, or network integrations must include ownership, threading, cancellation, late-callback, and shutdown tests.

Security

Muxiva is pre-alpha and must not be used to execute untrusted code or expose Studio directly to the public internet. Graph files must never contain secrets. Report vulnerabilities privately according to SECURITY.md.

See CHANGELOG.md for notable unreleased changes and SUPPORT.md for help channels and report requirements.

License

Muxiva is licensed under the Apache License 2.0.