Skip to content

README.md

Latest commit

 

History

History
335 lines (244 loc) · 16.6 KB

README.md

File metadata and controls

335 lines (244 loc) · 16.6 KB
Raw

OpenAdapt: AI-First Process Automation with Large Multimodal Models (LMMs)

Build Status PyPI version Downloads License: MIT Python 3.10+ Discord

OpenAdapt is the open source software adapter between Large Multimodal Models (LMMs) and traditional desktop and web GUIs.

Record a GUI workflow once, then compile it into a deterministic, self-healing replay that runs locally at near-zero cost — all from a unified CLI. OpenAdapt is a modular meta-package: pip install openadapt ships the flagship demonstration compiler (openadapt flow …) out of the box, and the supporting capabilities (capture, ML, evals, privacy) are optional extras you add as you need them.

Join us on Discord | Documentation | OpenAdapt.ai

Architecture

OpenAdapt v1.0+ uses a modular meta-package architecture. The main openadapt package provides a unified CLI and depends on focused sub-packages via PyPI:

Package Description Repository
openadapt Meta-package with unified CLI This repo
openadapt-capture Event recording and storage openadapt-capture
openadapt-ml ML engine, training, inference openadapt-ml
openadapt-evals Benchmark evaluation openadapt-evals
openadapt-flow Demonstration compiler (deterministic, self-healing replay) openadapt-flow
openadapt-viewer HTML visualization openadapt-viewer
openadapt-grounding UI element localization openadapt-grounding
openadapt-retrieval Multimodal demo retrieval openadapt-retrieval
openadapt-privacy PII/PHI scrubbing openadapt-privacy
openadapt-wright Dev automation openadapt-wright
openadapt-herald Social media from git history openadapt-herald
openadapt-crier Telegram approval bot openadapt-crier
openadapt-consilium Multi-model consensus openadapt-consilium
openadapt-desktop Desktop GUI application openadapt-desktop
openadapt-tray System tray app openadapt-tray
openadapt-agent Production execution engine openadapt-agent
openadapt-telemetry Error tracking openadapt-telemetry

Installation

Install what you need:

pip install openadapt              # CLI + demonstration compiler (openadapt flow …)
pip install openadapt[capture]     # + GUI capture/recording
pip install openadapt[ml]          # + ML training and inference
pip install openadapt[evals]       # + Benchmark evaluation
pip install openadapt[privacy]     # + PII/PHI scrubbing
pip install openadapt[all]         # Everything

The flagship demonstration compiler ships in the base install, so openadapt flow … works right after pip install openadapt.

Requirements: Python 3.10+

Quick Start

Record a workflow once, compile it into a deterministic bundle, and replay it locally at near-zero cost:

openadapt flow record --url <app> --out rec     # record a workflow once
openadapt flow compile rec --out bundle          # compile it
openadapt flow replay bundle                     # run it, local, $0

Inspect and gate compiled bundles before you ship them:

openadapt flow lint bundle                       # report coverage gaps
openadapt flow certify bundle --policy clinical-write   # enforce a safety policy

openadapt flow <verb> is the recommended path. The standalone openadapt-flow <verb> command keeps working and behaves identically.

Supporting commands

Capture, training, and evaluation are available once you install their extras (pip install openadapt[capture,ml,evals]):

openadapt capture start --name my-task           # record raw GUI events
openadapt train start --capture my-task --model qwen3vl-2b
openadapt eval run --checkpoint training_output/model.pt --benchmark waa
openadapt capture view my-task

Demonstration Compiler

For workflows you run over and over, re-reasoning through every step with a large model is slow, expensive, and non-deterministic. openadapt-flow compiles a single demonstration into a script that replays deterministically and locally, with no model calls on the hot path. It ships in the base openadapt install as the flagship path, so openadapt flow … works out of the box:

pip install openadapt
openadapt flow record --url <app> --out rec
openadapt flow compile rec --out bundle --name my-flow
openadapt flow replay bundle

It also ships standalone on PyPI (pip install openadapt-flow, currently v0.3.x); the standalone openadapt-flow <verb> command behaves identically to openadapt flow <verb>.

Each compiled step carries a template crop, an OCR label, geometry landmarks, and postconditions derived from what the demo changed on screen. At replay time a resolution ladder tries them in order (local match, global match, OCR, landmark geometry, then optionally a grounding model), so healthy runs cost milliseconds. When the UI drifts, a lower rung re-resolves the target and the fix lands back in the bundle as a reviewable diff — self-healing without a human in the loop. When the screen stops matching expectations, the run halts with a report instead of guessing, and identity-verified steps (for example a wrong-record check) refuse to act on a low-confidence match rather than click the wrong target.

The reference backend is a headless browser, which is why the whole loop runs in CI with no OS permissions; desktop and RDP backends are adapters in progress, not yet production paths. Compiled workflows can also be emitted as Agent Skills or MCP servers so other agents can invoke them.

In one field test against a computer-use agent on a real third-party EMR (OpenEMR's public demo), compiled replay matched the agent's success (20/20 compiled vs 10/10 agent) at roughly half the median latency and near-zero marginal cost — the agent cost about $0.55 per run, the compiled replay makes zero model calls. This is a small-sample result on a shared, daily-resetting public demo, so it is not CI-reproducible; a CI-reproducible control and the adversarial safety measurements are published alongside it.

Model-free on the hot path, deterministic, self-healing under drift, and honest about what it can't resolve. See openadapt-flow for the compiler, validation methodology, and known limits.

Ecosystem

Core Platform Components

Package Description Repository
openadapt Meta-package with unified CLI This repo
openadapt-capture Event recording and storage openadapt-capture
openadapt-ml ML engine, training, inference openadapt-ml
openadapt-evals Benchmark evaluation openadapt-evals
openadapt-flow Demonstration compiler (deterministic, self-healing replay) openadapt-flow
openadapt-viewer HTML visualization openadapt-viewer
openadapt-grounding UI element localization openadapt-grounding
openadapt-retrieval Multimodal demo retrieval openadapt-retrieval
openadapt-privacy PII/PHI scrubbing openadapt-privacy

Applications and Tools

Package Description Repository
openadapt-desktop Desktop GUI application openadapt-desktop
openadapt-tray System tray app openadapt-tray
openadapt-agent Production execution engine openadapt-agent
openadapt-wright Dev automation openadapt-wright
openadapt-herald Social media from git history openadapt-herald
openadapt-crier Telegram approval bot openadapt-crier
openadapt-consilium Multi-model consensus openadapt-consilium
openadapt-telemetry Error tracking openadapt-telemetry

CLI Reference

openadapt flow record --url <app> --out <dir>   Record a workflow once
openadapt flow compile <rec> --out <bundle>      Compile a recording into a bundle
openadapt flow replay <bundle>                   Replay a bundle (local, $0)
openadapt flow lint <bundle>                      Report a bundle's coverage gaps
openadapt flow certify <bundle> --policy <name>   Enforce a safety policy on a bundle

openadapt capture start --name <name>    Start recording
openadapt capture stop                    Stop recording
openadapt capture list                    List captures
openadapt capture view <name>             Open capture viewer

openadapt train start --capture <name>    Train model on capture
openadapt train status                    Check training progress
openadapt train stop                      Stop training

openadapt eval run --checkpoint <path>    Evaluate trained model
openadapt eval run --agent api-claude     Evaluate API agent
openadapt eval mock --tasks 10            Run mock evaluation

openadapt serve --port 8080               Start dashboard server
openadapt version                         Show installed versions
openadapt doctor                          Check system requirements

How It Works

See the full Architecture Evolution for detailed documentation.

Three-Phase Pipeline

OpenAdapt follows a streamlined Demonstrate → Learn → Execute pipeline:

1. DEMONSTRATE (Observation Collection)

  • Capture: Record user actions and screenshots with openadapt-capture
  • Privacy: Scrub PII/PHI from recordings with openadapt-privacy
  • Store: Build a searchable demonstration library

2. LEARN (Policy Acquisition)

  • Retrieval Path: Embed demonstrations, index them, and enable semantic search
  • Training Path: Load demonstrations and fine-tune Vision-Language Models (VLMs)
  • Abstraction: Progress from literal replay to template-based automation

3. EXECUTE (Agent Deployment)

  • Observe: Take screenshots and gather accessibility information
  • Policy: Use demonstration context to decide actions via VLMs (Claude, GPT-4o, Qwen3-VL)
  • Ground: Map intentions to specific UI coordinates with openadapt-grounding
  • Act: Execute validated actions with safety gates
  • Evaluate: Measure success with openadapt-evals and feed results back for improvement

Core Approach: Trajectory-Conditioned Disambiguation

Zero-shot VLMs fail on GUI tasks not due to lack of capability, but due to ambiguity in UI affordances. OpenAdapt resolves this by conditioning agents on human demonstrations — "show, don't tell."

No Retrieval With Retrieval
No Fine-tuning 46.7% (zero-shot baseline) 100% first-action (n=45, shared entry point)
Fine-tuning Standard SFT (baseline) Demo-conditioned FT (planned)

The bottom-right cell is OpenAdapt's unique value: training models to use demonstrations they haven't seen before, combining retrieval with fine-tuning for maximum accuracy. Phase 2 (retrieval-only prompting) is validated; Phase 3 (demo-conditioned fine-tuning) is in progress.

Validated result: On a controlled macOS benchmark (45 System Settings tasks sharing a common navigation entry point), demo-conditioned prompting improved first-action accuracy from 46.7% to 100%. A length-matched control (+11.1 pp only) confirms the benefit is semantic, not token-length. See the research thesis for methodology and the publication roadmap for limitations.

Industry validation: OpenCUA (NeurIPS 2025 Spotlight, XLANG Lab) reused OpenAdapt's macOS accessibility capture code in their AgentNetTool, but uses demos only for model training — not runtime conditioning. No open-source CUA framework currently does demo-conditioned inference, which remains OpenAdapt's architectural differentiator.

Key Concepts

  • Policy/Grounding Separation: The Policy decides what to do; Grounding determines where to do it
  • Safety Gate: Runtime validation layer before action execution (confirm mode for high-risk actions)
  • Abstraction Ladder: Progressive generalization from literal replay to goal-level automation
  • Evaluation-Driven Feedback: Success traces become new training data

Terminology

Term Description
Observation What the agent perceives (screenshot, accessibility tree)
Action What the agent does (click, type, scroll, etc.)
Trajectory Sequence of observation-action pairs
Demonstration Human-provided example trajectory
Policy Decision-making component that maps observations to actions
Grounding Mapping intent to specific UI elements (coordinates)

Demos

Legacy Version (v0.46.0) Examples:

Note: These demos show the legacy monolithic version. For current v1.0+ modular architecture examples, see the documentation.

Permissions

macOS: Grant Accessibility, Screen Recording, and Input Monitoring permissions to your terminal. See permissions guide.

Windows: Run as Administrator if needed for input capture.

Legacy Version

The monolithic OpenAdapt codebase (v0.46.0) is preserved in the legacy/ directory.

To use the legacy version:

pip install openadapt==0.46.0

See docs/LEGACY_FREEZE.md for migration guide and details.

Contributing

  1. Join Discord
  2. Pick an issue from the relevant sub-package repository
  3. Submit a PR

For sub-package development:

git clone https://github.com/OpenAdaptAI/openadapt-ml  # or other sub-package
cd openadapt-ml
pip install -e ".[dev]"

Related Projects

Support

License

MIT License - see LICENSE for details.