Skip to content

Latest commit

 

History

History
556 lines (393 loc) · 20.9 KB

development.md

File metadata and controls

556 lines (393 loc) · 20.9 KB

Development Guide

Guide for contributing to AgentTeams, building images locally, and running tests.

Prerequisites

  • Docker (for building and testing)
  • Git
  • mc (MinIO Client) for running integration tests
  • jq for JSON processing in test scripts

Project Structure

See ../../AGENTS.md for a comprehensive codebase navigation guide.

Runtime Harness checkouts

OpenClaw Worker source is maintained in agentteams-harness-openclaw and pinned here as the worker/ Git submodule. make build-worker and make push-worker initialize a missing checkout automatically. Update the harness first, then bump the submodule commit to upgrade the Worker packaging. Manager bootstrap, openclaw-base/, platform plugins and role templates remain in this repository.

Hermes runtime source is maintained in agentteams-harness-hermes and pinned here as the hermes/ Git submodule. Initialize it after cloning or pulling changes:

git submodule update --init --recursive

make build-hermes-worker initializes a missing checkout automatically. To upgrade Hermes, update the harness first, then bump the submodule commit and HERMES_GIT_REF in hack/versions.env together. Helm image defaults remain in this repository; installer defaults are maintained in the website repository.

Building Images Locally

From Core 1.2.7, Harness image repositories use these names:

Runtime Image repository basename
OpenClaw agentteams-harness-openclaw
QwenPaw agentteams-harness-qwenpaw
Hermes agentteams-harness-hermes
DeepSeek Harness agentteams-harness-deepseek
OpenHuman agentteams-harness-openhuman
Codex agentteams-harness-codex

Core local builds use the agentteams/ prefix. The v1.2.7 release publishes all six Harness images with the same v1.2.7 tag for amd64 and arm64 through .github/workflows/publish-harnesses.yml. Codex source remains in its independent repository. Manager roles reuse Harness images; dedicated Manager images are no longer published. Local Manager compatibility build targets remain available. Container names, runtime IDs and image override variables are unchanged. Installation scripts and their compatibility mappings are maintained in the website repository.

All builds go through the root Makefile:

# Build both Manager and Worker images (native arch, for local dev/test)
make build

# Build only Manager
make build-manager

# Build only Worker
make build-worker

# Controller + embedded all-in-one (infra + controller binary, no Manager agent)
make build-agentteams-controller
make build-embedded

# Alternate Manager / Worker runtimes
make build-manager-qwenpaw
make build-hermes-worker
make build-deepseek-harness-worker

# Build with a specific version tag
make build VERSION=local-dev

# Build a local DeepSeek Harness image with an explicit development tag
make build-deepseek-harness-worker VERSION=local-dev

# Build for a specific platform
make build DOCKER_PLATFORM=linux/amd64

See all available targets with make help.

Helm chart (helm/agentteams)

The production Kubernetes install is defined under helm/agentteams/ (subcharts for gateway, homeserver, storage, etc.). Useful Makefile targets:

make helm-lint      # helm dependency build + helm lint
make helm-template  # render templates locally (validation)
make sync-crds      # copy CRD YAML from agentteams-controller/config/crd/ into the chart

Push Images (Multi-Architecture by Default)

make push always builds multi-arch manifests (amd64 + arm64) to avoid accidentally overwriting multi-arch images with a single-arch image. This uses docker buildx:

# Build amd64 + arm64 and push to registry
make push VERSION=0.1.0 REGISTRY=ghcr.io REPO=agentscope-ai/AgentTeams

# Push only Manager (multi-arch)
make push-manager VERSION=latest

# Customize platforms (default: linux/amd64,linux/arm64)
make push MULTIARCH_PLATFORMS=linux/amd64,linux/arm64,linux/arm/v7

Note: make push always pushes directly to the registry (required by buildx multi-platform builds — multi-arch images cannot be stored in the local Docker image store). You must docker login to the target registry first.

For local development and testing, use make build which creates native-arch images locally. If you absolutely need to push single-arch images (not recommended), use make push-native.

Regional Registry Mirrors

All base images are hosted on the Higress Alibaba Cloud registry with regional mirrors that auto-sync from the primary (cn-hangzhou). Choose the nearest region for faster pulls:

Region Registry Usage
China (default) higress-registry.cn-hangzhou.cr.aliyuncs.com Default, no extra args needed
North America higress-registry.us-west-1.cr.aliyuncs.com make build HIGRESS_REGISTRY=higress-registry.us-west-1.cr.aliyuncs.com
Southeast Asia higress-registry.ap-southeast-7.cr.aliyuncs.com make build HIGRESS_REGISTRY=higress-registry.ap-southeast-7.cr.aliyuncs.com

Or pass it directly to Docker:

docker build --build-arg HIGRESS_REGISTRY=higress-registry.us-west-1.cr.aliyuncs.com -t agentteams/manager:latest ./manager/

Install / Uninstall / Replay

Quick Install (Minimal)

Only AGENTTEAMS_LLM_API_KEY is required — everything else uses sensible defaults:

# Build images + install Manager with one command
AGENTTEAMS_LLM_API_KEY="sk-xxx" make install

This will:

  1. Build Manager and Worker images (make build)
  2. Run the install script (bash <(curl -fsSL https://agentteams.cloud/install.sh) manager)
  3. Mount the container runtime socket for direct Worker creation
  4. Save configuration to ./agentteams-manager.env

Custom Install

Override any configuration via environment variables:

AGENTTEAMS_LLM_API_KEY="sk-xxx" \
AGENTTEAMS_LLM_PROVIDER="openai" \
AGENTTEAMS_DEFAULT_MODEL="gpt-4o" \
AGENTTEAMS_ADMIN_USER="myadmin" \
AGENTTEAMS_ADMIN_PASSWORD="mypassword" \
make install

Uninstall

make uninstall   # Runs bash <(curl -fsSL https://agentteams.cloud/install.sh) uninstall (v1.1+: controller + manager + workers + volume + env)

Replay (Send Task to Manager)

After installing, send tasks to the Manager via the Matrix protocol:

# CLI mode: pass task as argument
make replay TASK="Create a Worker named alice for frontend development"

# Interactive mode: prompts for input
make replay

# Pipe mode: read from stdin
echo "Create worker bob" | ./scripts/replay-task.sh

The replay script:

  • Reads credentials from ./agentteams-manager.env
  • Logs into Matrix as the admin user
  • Finds (or auto-creates) the DM room with the Manager
  • Sends the task message
  • Waits for and prints the Manager's reply (configurable via REPLAY_WAIT=0 to skip)
  • Saves conversation logs to logs/replay/replay-{timestamp}.log (view with make replay-log)

Test Against Installed Manager

After make install, run the test suite against the running Manager without rebuilding or creating a new container:

make test-installed

# Or with a test filter
make test-installed TEST_FILTER="01 02"

This reads credentials from ./agentteams-manager.env and skips the container lifecycle (start/stop).

Running Tests

Full Integration Test Suite

# Build images + run all 10 test cases
export AGENTTEAMS_LLM_API_KEY="your-api-key"
make test

Run Specific Tests

# Run only tests 01, 02, 03
make test TEST_FILTER="01 02 03"

Skip Image Build

# Use existing images (faster iteration)
make test SKIP_BUILD=1

Quick Smoke Test

# Run test-01 only (quick health check)
make test-quick

GitHub Tests (08-10)

Tests 08-10 require a GitHub Personal Access Token:

export AGENTTEAMS_GITHUB_TOKEN="ghp_..."
make test TEST_FILTER="08 09 10"

Making Changes

Modifying Agent Behavior

Agent behavior is defined by markdown files, not code:

  • Manager SOUL: manager/agent/SOUL.md
  • Manager Heartbeat: manager/agent/HEARTBEAT.md
  • Skills: manager/agent/skills/*/SKILL.md

Modifying Startup Sequence

Each component has its own startup script in manager/scripts/init/:

  • Modify the relevant start-*.sh script
  • Rebuild the Manager image
  • Run tests to verify

Modifying Higress Configuration

Route, consumer, and MCP server bootstrap for embedded stacks is owned by the controller image (agentteams-controller / Dockerfile.embedded wiring). The legacy manager/scripts/init/setup-higress.sh path still applies to older single-container Manager images (≤v1.0.9) and to Manager-side behaviors where applicable — check your install mode before editing.

Adding a New MCP Server

  1. Add the server configuration to setup-higress.sh
  2. Create a Worker skill SKILL.md documenting available tools
  3. Update manager/agent/worker-agent/skills/ with the new skill
  4. Add test coverage in tests/

CI/CD

Parallel release images

build.yml runs each image on a separate runner. OpenClaw Base, Controller, and QwenPaw Worker start independently. Embedded, QwenPaw Manager, and Hermes Worker wait for Controller; OpenClaw Manager and Worker wait for both Base and Controller. Each image still builds amd64 and arm64 using QEMU.

Before building, each job inspects its version tag. If both linux/amd64 and linux/arm64 already exist, it succeeds without building or pushing; downstream jobs continue normally. Only an explicit image-not-found response or a valid manifest missing a required architecture triggers a build. Query failures (including authentication, rate limiting, and network errors) or invalid responses fail the job without rebuilding. Reuse is based on the tag and architectures, not source commit identity, and also applies to latest. Reusing a version does not update the latest alias. Use a new version to build changed source.

A version tag starts the full build, then calls release.yml only after every image job succeeds. Release verifies all nine versioned multi-architecture manifests before publishing. There is no fixed polling window while images are building.

For manual builds, select Build Images, set version, and set targets to all or exact space-separated target names. Required Base/Controller dependencies are built automatically. Manual builds do not publish a GitHub Release. To publish manually, first finish the full build, then run Release with the same version and source ref. If publication fails after the images succeed, rerun the failed release job.

Validate workflow changes locally with actionlint and python3 tests/test-build-workflows.py (requires PyYAML and jq).

GitHub Actions Workflows

Workflow Trigger Purpose Arch
build.yml Version tags / manual Parallel multi-arch build + push amd64 + arm64
integration-test.yml After build succeeds on main Run full test suite amd64 (runner native)
release.yml After tag build succeeds / manual Verify images and publish GitHub Release amd64 + arm64

All CI multi-arch builds use docker/setup-qemu-action for cross-platform emulation and docker buildx via make push.

Required Secrets

Secret Purpose
AGENTTEAMS_LLM_API_KEY LLM access for Agent behavior tests
AGENTTEAMS_GITHUB_TOKEN GitHub operations in tests 08-10

Local CI Simulation

# Same flow as CI but local (single arch)
export AGENTTEAMS_LLM_API_KEY="your-key"
make test

# Multi-arch build like CI does on main branch
docker login ghcr.io
make push VERSION=latest REGISTRY=ghcr.io REPO=agentscope-ai/AgentTeams

Network Proxy Setup (China Mainland)

Building images requires accessing GitHub (for git clone) and npm registries. In China mainland environments, a proxy is typically needed.

Host Proxy

Enable proxy in your shell before running commands:

# Enable proxy (adjust ports to your proxy config)
export http_proxy="http://127.0.0.1:1087"
export https_proxy="http://127.0.0.1:1087"
export ALL_PROXY="socks5://127.0.0.1:1086"

# IMPORTANT: exclude localhost from proxy, otherwise test health checks fail
export no_proxy="localhost,127.0.0.1,::1,local,169.254/16"

Docker Build Proxy

Docker build runs in an isolated environment and does NOT inherit host proxy settings. Pass proxy via DOCKER_BUILD_ARGS:

make build-manager DOCKER_BUILD_ARGS="--build-arg http_proxy=http://host.docker.internal:1087 --build-arg https_proxy=http://host.docker.internal:1087"

Note: host.docker.internal resolves to the host machine from within Docker containers. If your proxy listens on 127.0.0.1:1087 on the host, use host.docker.internal:1087 in build args.

Running Tests with Proxy

When running tests, no_proxy is critical — without it, test health checks to 127.0.0.1 go through the proxy and return 503:

export no_proxy="localhost,127.0.0.1,::1,local,169.254/16"
AGENTTEAMS_LLM_API_KEY="your-key" make test SKIP_BUILD=1

Container Runtime Socket (Direct Worker Creation)

When the Manager container is started with the host's container runtime socket mounted, it can create Worker containers directly — no human intervention needed for local deployments.

How It Works

The Manager detects the socket at startup and sets AGENTTEAMS_CONTAINER_RUNTIME=socket. The container-api.sh script provides functions to create/start/stop Worker containers via the Docker-compatible REST API (works with both Docker and Podman).

Socket Paths

Runtime Socket path (host) Mount command
Docker /var/run/docker.sock -v /var/run/docker.sock:/var/run/docker.sock
Podman (rootful, Linux) /run/podman/podman.sock -v /run/podman/podman.sock:/var/run/docker.sock --security-opt label=disable
Podman (macOS machine) Inside VM: /run/podman/podman.sock Same as rootful (VM provides symlink at /var/run/docker.sock)

Example: Manual Start with Socket

# Docker
docker run -d --name agentteams-manager \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e AGENTTEAMS_WORKER_IMAGE=agentteams/agentteams-harness-openclaw:latest \
  ... \
  agentteams/manager:latest

# Podman
podman run -d --name agentteams-manager \
  -v /run/podman/podman.sock:/var/run/docker.sock \
  --security-opt label=disable \
  -e AGENTTEAMS_WORKER_IMAGE=agentteams/agentteams-harness-openclaw:latest \
  ... \
  agentteams/manager:latest

Test Integration

The test orchestrator (tests/run-all-tests.sh) automatically detects the socket and mounts it when available.

Security Note

Mounting the container runtime socket gives the container full control over the host's container runtime (equivalent to root access). This is acceptable for local development. In production, consider using more restrictive approaches like Podman socket activation with limited API access.

Key Technical Notes

Node.js Version

OpenClaw requires Node.js >= 22 (the --disable-warning flag used internally requires Node.js 21.3+). The Manager image is built on openclaw-base which already includes Node.js 24. The Worker Dockerfile copies Node.js 24 from a build stage.

  • Manager: The latest Node 24 LTS (floor 24.22.1) is provided by openclaw-base (resolved and SHA-verified at build time).
  • Node overrides: NODE_VERSION selects a 24.x patch (≥24.22.1); the major is fixed to 24, so 25.x/26.x overrides are rejected even when NODE_MAJOR=24 is passed.
  • Worker: The latest Node 24 LTS (floor 24.22.1) binary copied from build stage replaces Ubuntu 24.04 apt's Node.js 18.x (which lacks --disable-warning support).

Higress AI Provider API

When creating a qwen type provider via the Higress Console API, you must include rawConfigs with Qwen-specific fields — otherwise the API returns "Missing Qwen specific configurations". The correct body is:

{
  "type": "qwen",
  "name": "qwen",
  "tokens": ["your-api-key"],
  "protocol": "openai/v1",
  "rawConfigs": {
    "qwenEnableSearch": false,
    "qwenEnableCompatible": true,
    "qwenFileIds": []
  }
}

For OpenAI-compatible providers (DeepSeek, etc.), use type=openai with rawConfigs.apiUrl pointing to the provider's endpoint. This is all handled automatically in manager/scripts/init/setup-higress.sh.

OpenClaw Skills Format

SKILL.md files must include a YAML front matter block, otherwise OpenClaw will not discover them:

---
name: my-skill-name
description: What this skill does and when to use it
---

# Skill Title
...content...

Skills placed in <workspace>/skills/<name>/SKILL.md are auto-discovered (source: openclaw-workspace).

OpenClaw Gateway Config

The openclaw.json must include gateway configuration for headless operation:

{
  "gateway": {
    "mode": "local",
    "port": 18799,
    "auth": { "token": "<some-token>" }
  }
}

Without gateway.mode=local, OpenClaw refuses to start. Without gateway.auth.token, it also refuses.

Code Style

  • Shell scripts: use ${VAR} syntax, functions for reusable logic
  • Config templates: ${VAR} placeholders, comments explaining each field
  • Skills (SKILL.md): Must include YAML front matter (name + description), self-contained with full API reference and examples
  • Tests: One file per acceptance case, source shared helpers, use assertion functions

Debugging Tips

View Manager Logs

Container name is agentteams-manager (via make install) or agentteams-manager-test (via make test).

v1.1.0+ embedded install: infrastructure logs are on agentteams-controller, not the Manager container.

# Manager Agent
docker exec agentteams-manager cat /var/log/agentteams/manager-agent.log
docker exec agentteams-manager cat /var/log/agentteams/manager-agent-error.log  # OpenClaw gateway stderr (OpenClaw Manager)
docker exec agentteams-manager bash -c 'cat /tmp/openclaw/openclaw-*.log' | jq .

# Higress / homeserver (embedded controller)
docker exec agentteams-controller cat /var/log/agentteams/higress-console.log
docker exec agentteams-controller cat /var/log/agentteams/tuwunel.log

View Replay Conversation Logs

# After running `make replay`, logs are saved automatically
make replay-log

# Logs directory: logs/replay/replay-{timestamp}.log

Check OpenClaw Skills Loading

docker exec agentteams-manager bash -c \
  'OPENCLAW_CONFIG_PATH=/root/manager-workspace/openclaw.json openclaw skills list --json' \
  | jq '.skills[] | select(.source == "openclaw-workspace") | {name, eligible, description}'

Interactive Shell in Container

docker exec -it agentteams-manager bash

Check Higress State

# Login (init uses "name", login uses "username")
curl -X POST http://localhost:18001/session/login \
  -H 'Content-Type: application/json' \
  -c /tmp/cookie \
  -d '{"username": "admin", "password": "your-password"}'

# List consumers
curl -s http://localhost:18001/v1/consumers -b /tmp/cookie | jq

# List routes
curl -s http://localhost:18001/v1/routes -b /tmp/cookie | jq

# List AI providers
curl -s http://localhost:18001/v1/ai/providers -b /tmp/cookie | jq

# List AI routes
curl -s http://localhost:18001/v1/ai/routes -b /tmp/cookie | jq

Check MinIO State

mc alias set test http://localhost:9000 <user> <password>
mc ls test/agentteams-storage/ --recursive

Common Issues

Symptom Cause Fix
git clone hangs during docker build No proxy in build env Pass --build-arg http_proxy=... via DOCKER_BUILD_ARGS
Health checks return 503 http_proxy capturing localhost requests Set no_proxy=localhost,127.0.0.1,::1
OpenClaw: SyntaxError: Unexpected reserved word Node.js too old Ensure Manager uses openclaw-base image; Worker uses Node.js 24 from build stage
OpenClaw: requires Node >=22.0.0 Same as above Same as above
--disable-warning= is not allowed in NODE_OPTIONS Node.js < 21.3 (e.g., Ubuntu apt's v18) Ensure Worker uses Node.js 24 from build stage, not apt
OpenClaw: gateway.mode=local required Missing gateway config in openclaw.json Add "gateway": {"mode": "local", ...}
OpenClaw: no token is configured Missing gateway auth token Add "gateway": {"auth": {"token": "..."}}
Higress: Missing Qwen specific configurations type=qwen requires rawConfigs fields Include rawConfigs: {qwenEnableCompatible: true, ...} — see setup-higress.sh
Skills not loaded by OpenClaw Missing YAML front matter in SKILL.md Add ---\nname: ...\ndescription: ...\n---
setup-higress.sh crashes on restart set -e + curl -sf on "already exists" errors Use higress_api() helper which handles this gracefully
Higress setup runs again on restart, resets consumers Missing setup marker Check /data/.higress-setup-done; delete it only to force re-setup