architecture.md

AbstractCore Architecture

AbstractCore provides a unified interface to major LLM providers with production-oriented reliability features. This document explains how it works internally and why it's designed this way.

If you're new to AbstractCore and want to start building quickly, read:

• docs/getting-started.md
• docs/api.md

Related docs (user-facing):

• Media inputs (images/audio/video + documents): docs/media-handling-system.md
• Vision input + fallback: docs/vision-capabilities.md
• Capability plugins (voice/audio/vision): docs/capabilities.md
• OpenAI-compatible gateway server: docs/server.md
• Single-model OpenAI-compatible endpoint: docs/endpoint.md
• Tool calling semantics (passthrough vs execution): docs/tool-calling.md
• Local models, engines and download jobs: docs/models.md, docs/engines.md, docs/console.md

System Overview

AbstractCore operates as a Python library and can also be exposed via optional OpenAI-compatible HTTP servers:

• Gateway server (multi-provider): abstractcore.server.app (docs: docs/server.md)
• Endpoint server (single-model): abstractcore.endpoint.app (docs: docs/endpoint.md)

graph TD
    A[Your Application] --> B[AbstractCore API]
    AA[HTTP Clients] --> BB[AbstractCore Server]
    BB --> B
    
    B --> C[Provider Interface]
    C --> D[Event System]
    C --> E[Tool System]
    C --> F[Retry System]
    C --> G[Provider Implementations]

    G --> H[OpenAI Provider]
    G --> HH[OpenAI-Compatible Provider]
    G --> I[Anthropic Provider]
    G --> J[Ollama Provider]
    G --> K[MLX Provider]
    G --> L[LMStudio Provider]
    G --> M[HuggingFace Provider]
    G --> MM[vLLM Provider]
    G --> MN[OpenRouter Provider]
    G --> MP[Portkey Provider]

    H --> N[OpenAI API]
    HH --> NN[OpenAI-Compatible /v1 Endpoint]
    I --> O[Anthropic API]
    J --> P[Ollama Server]
    K --> Q[MLX Models]
    L --> R[LMStudio Server]
    M --> S[HuggingFace Models]
    MM --> RR[vLLM Server]
    MN --> RO[OpenRouter API]
    MP --> RP[Portkey API Gateway]

    style B fill:#e1f5fe
    style BB fill:#4caf50
    style C fill:#f3e5f5
    style G fill:#fff3e0

Loading

Design Principles

1. Provider Abstraction

Goal: Same interface for all providers Implementation: Common interface with provider-specific implementations

2. Production Reliability

Goal: Handle real-world failures gracefully Implementation: Built-in retry logic, circuit breakers, comprehensive error handling

3. Universal Tool Support

Goal: Tools work everywhere, even with providers that don't support them natively Implementation: Native support where available, intelligent prompting as fallback

4. Simplicity Over Features

Goal: Clean, focused API that's easy to understand Implementation: Minimal core with clear extension points

5. Optional HTTP Access

Goal: Flexible deployment as library or server Implementation: OpenAI-compatible REST API built on core library

Core Components

Native MLX execution ownership

The opt-in mlx_batching=True runtime serves the documented Qwen3.8-27B and Flash-Next checkpoints in one process. Ordinary MLX execution and its cache artifact format remain separate. No oMLX component is required.

flowchart TD
    PY[Python sync / async / streaming callers] --> P[MLXProvider: request-local normalization]
    HTTP[HTTP chat clients] --> API[AbstractCore server or single-model endpoint]
    API --> P
    CONTRACT[Shared speculation contract: controls and outcomes] -.-> P
    P --> Q[Immutable requests / bounded admission queue]
    Q --> W[One execution worker per shared native session]
    SESSION[NativeSession: target, processor, MTP head, owner leases] --> W
    W --> POLICY{Compatible request policy}
    POLICY --> TARGET[Greedy target-only: continuous admission]
    POLICY --> MTP[Greedy MTP: fixed same-depth cohort]
    POLICY --> EXCLUSIVE[Sampled / incompatible controls: exclusive]
    TARGET --> ENGINE[mlx-vlm / mlx-lm on MLX Metal]
    MTP --> ENGINE
    EXCLUSIVE --> ENGINE
    ENGINE <--> CACHE[Bounded RAM prefix cache / optional private SSD]
    ENGINE --> OUT[Per-request bounded output / usage / execution metadata]
    OUT --> P
    CONTROL[Load / cache / unload control plane] -. serialized operations .-> SESSION

Loading

Component responsibilities:

Component	Responsibility
providers/speculation.py	Backend-independent request validation and observed-outcome types; no tensor execution
MLXProvider	Public API, request-local thinking/tool/sampling controls, media handling and response normalization
mlx_native_session.py	Shared target/head/processor identity and provider leases; sibling-safe lifecycle
mlx_runtime.py	One execution owner, admission policies, bounded queues, cancellation and truthful per-request accounting
mlx_native_cache.py	Bounded native prefix reuse, cache identity, private single-writer SSD persistence and owned-entry clearing
mlx_qwen4.py	Flash-Next loading, embedded head, PLE offload and architecture-specific cache compatibility
Upstream MLX / mlx-lm / mlx-vlm	Model graphs, image processing, tensor kernels, draft verification and cache rollback
HTTP adapters / async stream bridge	Concurrent generation admission, backpressure, disconnect propagation and owner-thread iterator cleanup

Request flow:

1. Normalize the public request once, render its complete context and retain request-local controls. Image requests carry actual pixels to the native processor/vision encoder; MTP off does not disable vision.
2. Bind the request to its shared session and enqueue immutable execution data. The worker prepares text/image embeddings and looks up an eligible prefix.
3. Select continuous target decoding, a fixed MTP cohort, or exclusive execution. MTP retains target hidden state, drafts num_draft_tokens proposals, verifies them with the target, accepts a prefix and reconciles rejected cache state.
4. Stream accepted output through the request's bounded channel. Final usage counts request tokens; shared cohort draft counters remain separate diagnostics.
5. Cancel at a safe backend boundary, or finish and retain an eligible prefix. Final-owner unload flushes storage and releases weights; an idle sibling can release its lease without unloading another owner's model.

The 27B checkpoint uses a matching separate trained MTP head; Flash-Next embeds its head. Their ordinary target weights are still the verifier. Model-level MTP support requires both trained head tensors and a backend that executes them; the shared contract alone does not supply HuggingFace/GGUF MTP.

See runtime configuration and boundaries, speculative decoding, and measured scheduling tradeoffs.

1. Factory Pattern (create_llm)

The main entry point uses the factory pattern for clean provider instantiation:

graph LR
    A[create_llm] --> B{Provider Type}
    B --> C[OpenAI Provider]
    B --> D[Anthropic Provider]
    B --> E[Ollama Provider]
    B --> F[Other Providers...]

    C --> G[Configured Instance]
    D --> G
    E --> G
    F --> G

    style A fill:#4caf50
    style G fill:#2196f3

Loading

from abstractcore import create_llm

# Factory creates the right provider with proper configuration
llm = create_llm("openai", model="gpt-4o-mini", temperature=0.7)

# OpenAI-compatible /v1 endpoints (LMStudio, vLLM, custom proxies)
llm_local = create_llm("lmstudio", model="qwen/qwen3-4b-2507", base_url="http://localhost:1234/v1")
llm_openrouter = create_llm("openrouter", model="openai/gpt-4o-mini")  # requires OPENROUTER_API_KEY
llm_portkey = create_llm("portkey", model="gpt-4o-mini", config_id="pcfg_...")  # requires PORTKEY_API_KEY + PORTKEY_CONFIG

Gateway providers (OpenRouter/Portkey) route to external backends; AbstractCore forwards only explicit generation parameters to avoid sending defaults that strict backends reject.

2. Provider Interface

All providers implement AbstractCoreInterface (see abstractcore/core/interface.py):

class AbstractCoreInterface(ABC):
    @abstractmethod
    def generate(
        self,
        prompt: str,
        messages: Optional[List[Dict[str, str]]] = None,
        system_prompt: Optional[str] = None,
        tools: Optional[List[Dict[str, Any]]] = None,
        media: Optional[List[Union[str, Dict[str, Any], "MediaContent"]]] = None,
        stream: bool = False,
        thinking: Optional[Union[bool, str]] = None,
        **kwargs,
    ) -> Union[GenerateResponse, Iterator[GenerateResponse]]:
        """Generate a response (or a stream of chunks)."""

    @abstractmethod
    def get_capabilities(self) -> List[str]:
        """Get provider capabilities"""

    @abstractmethod
    def unload_model(self, model_name: str) -> None:
        """Unload/cleanup resources for a specific model (best-effort)."""

This ensures:

• Consistency: Same methods across all providers
• Reliability: Standardized error handling
• Extensibility: Easy to add new providers
• Memory Management: Explicit control over model lifecycle

Response Normalization (Model Output Cleanup)

BaseProvider also applies asset-driven response normalization so downstream code sees clean, consistent output across providers:

• Output wrappers: Strip configured leading/trailing wrapper tokens (e.g., GLM <|begin_of_box|>…<|end_of_box|>)
• Harmony transcripts (GPT-OSS): Extract <|channel|>final into GenerateResponse.content and capture <|channel|>analysis as GenerateResponse.metadata["reasoning"] (non-streaming)
• Thinking tags: Extract inline <think>...</think> blocks into GenerateResponse.metadata["reasoning"] (when configured)

Why this belongs in BaseProvider (even for streaming):

• These artifacts are model/template-specific, not provider-specific (the same model can be served via Ollama, vLLM, LMStudio, HF, or MLX)
• In streaming mode, wrappers often appear in the first/last chunks; stripping them incrementally avoids leaking markup into UIs and tool parsers without buffering the full response

Configuration comes from abstractcore/assets/architecture_formats.json and abstractcore/assets/model_capabilities.json; implementation lives in abstractcore/architectures/response_postprocessing.py.

Unified Thinking Control

thinking=... is a single public generation parameter, not a per-provider option. BaseProvider normalizes the request, applies model-family rules, and then calls each provider's optional _apply_provider_thinking_kwargs(...) hook. The hook returns rewritten kwargs plus a ThinkingControlHandling record so AbstractCore can tell whether enable/disable and effort-level control were actually handled.

Provider implementations should prefer native controls such as tokenizer enable_thinking, server chat_template_kwargs, or provider-native request fields. Prompt-marker fallbacks are only used when no cleaner control exists, and local cache renderers must apply the same thinking control during prompt_cache_update(...) and generate(...) so durable cache prefixes remain token compatible with live requests.

Model Metadata Registry (Source of Truth)

AbstractCore's model capability routing and architecture formatting are driven by two canonical JSON registries:

• abstractcore/assets/model_capabilities.json — model limits, tool/structured output flags, multimodal support, route-keyed capability_routes, aliases
• abstractcore/assets/model_capabilities.schema.json — JSON Schema contract for model capability metadata
• abstractcore/assets/architecture_formats.json — message formats, tool call syntax, response wrappers, detection patterns

When a new model or architecture is released (or an existing one changes), update these files first. See abstractcore/assets/README.md for field requirements and update rules.

Memory Management

The unload_model(model_name) method is a best-effort resource cleanup hook.

• API providers (OpenAI, Anthropic): typically a no-op (safe to call).
• Local / self-hosted providers: behavior is provider-specific:
  • some can actively release memory (or request server-side eviction),
  • others can only close client connections and rely on server-side TTL/auto-eviction.
  • Examples: Ollama uses native keep_alive load/unload semantics, and LM Studio uses its native loaded-instance REST API when available.

On in-process providers (MLX, HuggingFace), unloading also drops the instance's session prompt caches (KV cache store and MLX hybrid snapshots): they are only useful with the weights resident, and they are the memory hogs.

Provider/server availability and model catalog membership are not loaded-model proof. When Core can verify residency, providers expose get_model_residency(...); otherwise loaded state is reported as unknown/fail-closed. Instances list what they can verify via list_loaded_models(), and abstractcore.utils.residency.sweep_loaded_models() sweeps the host's local model servers (Ollama, LM Studio) best-effort. abstractcore.utils.memory.get_memory_snapshot() reports host RAM, process RSS, device allocation, and the host identity; device.process_held_bytes is the accelerator memory this process holds across MLX, torch and llama.cpp, and device.allocated_bytes verifies an MLX unload — not process RSS, which behaves as a high-water mark on Metal hosts. GGUF loads that settle their context through the probe ladder record the settled context to a per-machine calibration store, which seeds later loads and powers abstractcore.utils.context_estimate.estimate_context_fit() — an analytical context-fit answer that never loads weights. See Memory and Model Residency.

unload_model() releases one instance's hold. Several holders can keep the same model alive in one process (MLX instances share weights; HuggingFace and embedding instances each hold a full copy), so abstractcore.providers.process_residency gives one door to every in-process backend: resident_rows() lists every model alive in the process, eject() unloads every holder, and eject_unclaimed() ejects only when no registered owner (the server's managed runtimes, AbstractRuntime clients) still pools, locks or is loading the model. The claim check and the eject run under one process lock.

graph TD
    UNLOAD["POST /acore/models/unload<br/>or unload_after"] --> RT["Managed runtime:<br/>unload_model()"]
    RT --> EU["process_residency.eject_unclaimed()"]
    CLIENT["AbstractRuntime client<br/>default switch"] --> EU
    EU -->|"check under residency_lock()"| CLAIMS["Registered claimants:<br/>server managed runtimes,<br/>runtime client pools and locks"]
    CLAIMS -->|"model still claimed"| SKIP["Skipped report;<br/>HTTP 409 model_locked / model_in_use<br/>for a process-only unload"]
    CLAIMS -->|"no claim"| EJ["process_residency.eject()"]
    EJ --> MLX["mlx_residency.eject_model():<br/>every holder, drafter, KV caches,<br/>MLX allocator cache"]
    EJ --> HF["hf_residency.eject_model():<br/>transformers and GGUF instances,<br/>torch MPS pool"]
    EJ --> EMB["eject_embedding_models():<br/>in-process EmbeddingManagers<br/>(reload on next embed)"]
    MLX --> SNAP["get_memory_snapshot():<br/>device.process_held_bytes,<br/>resident block"]
    HF --> SNAP
    EMB --> SNAP

Loading

In the OpenAI-compatible AbstractCore server (abstractcore.server.app), requests can set unload_after (default false) to unload the model after the request completes; it uses the same process-wide eject as POST /acore/models/unload and skips locked runtimes. For providers that can unload shared server state (e.g. Ollama), this is disabled by default and must be explicitly enabled by the server operator.

# Load model, use it, then free memory
llm = create_llm("ollama", model="large-model")
response = llm.generate("Hello")
llm.unload_model(llm.model)  # Explicitly free memory
del llm

This is critical for:

• Test suites that load multiple models sequentially
• Memory-constrained environments (<32GB RAM)
• Production systems serving different models sequentially

3. Media Handling System

AbstractCore includes a policy-driven media handling system that enables file attachments across all providers:

graph TD
    A[User Input: @file.pdf] --> B[MessagePreprocessor]
    B --> C[Extract Files + Clean Text]
    C --> D[AutoMediaHandler]
    D --> E{File Type Detection}
    E -->|Images| F[ImageProcessor]
    E -->|PDFs| G[PDFProcessor]
    E -->|Office| H[OfficeProcessor]
    E -->|Text/CSV| I[TextProcessor]

    F --> J[MediaContent Objects]
    G --> J
    H --> J
    I --> J

    J --> K{Provider Type}
    K -->|OpenAI| L[OpenAI Format]
    K -->|Anthropic| M[Anthropic Format]
    K -->|Local| N[Text Embedding]

    L --> O[Provider API Call]
    M --> O
    N --> O

    style D fill:#4caf50
    style J fill:#2196f3
    style O fill:#ff9800

Loading

Media System Architecture

Core Components:

• MessagePreprocessor: Parses @filename syntax in CLI and extracts file references
• AutoMediaHandler: Intelligent coordinator that selects appropriate processors
• Specialized Processors:
  • ImageProcessor (PIL-based for images)
  • PDFProcessor (pypdf by default for permissive PDF text/metadata extraction; optional PyMuPDF4LLM backend is explicit opt-in)
  • OfficeProcessor (Unstructured for DOCX/XLSX/PPTX)
  • TextProcessor (pandas for CSV/TSV data analysis)
• Provider Handlers: Format media content for each provider's API requirements

Provider-Specific Formatting:

# Same MediaContent gets formatted differently:

# OpenAI (JSON with image_url):
{
  "role": "user",
  "content": [
    {"type": "text", "text": "Analyze this"},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
  ]
}

# Anthropic (Messages API with source):
{
  "role": "user",
  "content": [
    {"type": "text", "text": "Analyze this"},
    {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "..."}}
  ]
}

# Local (Text embedding):
"Analyze this\n\nImage description: A chart showing quarterly trends..."

Document Processing Strategy:

1. Specialized Processing: pypdf for PDFs by default, Unstructured for Office documents
2. Basic Processing: Simple text extraction
3. Metadata Fallback: File information and properties
4. Degrades gracefully for documents: PDFs/Office/text aim to return best-effort extracted text/metadata rather than crashing.
5. Policy-driven for true multimodal inputs: for image/audio/video message parts, behavior is policy-driven; unsupported requests fail loudly unless an explicit enrichment fallback is configured (see docs/media-handling-system.md and docs/vision-capabilities.md).

Unified Media API

The same media=[] parameter works across all providers:

# Universal API - works with any provider
llm = create_llm("openai", model="gpt-4o")  # or "anthropic", "ollama", etc.
response = llm.generate(
    "Analyze these files",
    media=["report.pdf", "chart.png", "data.xlsx"]
)

CLI Integration:

# Simple @filename syntax works everywhere
python -m abstractcore.utils.cli --prompt "What's in @document.pdf and @image.jpg"

Capability plugins (voice/audio/vision/music)

To keep the default abstractcore install dependency-light while still enabling deterministic modality APIs, AbstractCore supports optional capability plugins:

• abstractvoice provides core.voice + core.audio (TTS/STT).
• abstractvision provides core.vision (T2I/I2I/T2V/I2V; backend-pluggable).
• abstractmusic can provide core.music (text-to-music) when installed.

Discovery:

• llm.capabilities.status() returns a JSON-safe snapshot (which backends are available/selected, plus install hints).
• llm.capabilities.list_backend_infos() returns registered backend metadata without instantiating backend factories.
• llm.capabilities.available_providers(capability, task=...) and llm.capabilities.list_models(capability, task=..., provider=...) expose the shared discovery contract for plugins that implement it.
• Convenience facades exist as properties: llm.voice, llm.audio, llm.vision, and llm.music (lazy; missing plugins raise actionable errors).
• Plugins that need LLM text planning can use llm.capability_host_context.text; this is a text-only service wrapper, not access to the raw provider object.

4. Request Lifecycle

sequenceDiagram
    participant App as Your App
    participant Core as AbstractCore
    participant Events as Event System
    participant Retry as Retry Logic
    participant Provider as LLM Provider
    participant Tools as Tool System

    App->>Core: generate("prompt", tools=tools)
    Core->>Events: emit(GENERATION_STARTED)
    Core->>Retry: wrap_with_retry()

    alt Provider Call Success
        Retry->>Provider: API call
        Provider->>Retry: response
        Retry->>Core: successful response
    else Provider Call Fails
        Retry->>Provider: API call (attempt 1)
        Provider->>Retry: rate limit error
        Retry->>Retry: wait with backoff
        Retry->>Provider: API call (attempt 2)
        Provider->>Retry: success
        Retry->>Core: successful response
    end

    alt Has Tool Calls
        Core->>Events: emit(TOOL_STARTED)
        Core->>Tools: execute_tools()
        Tools->>Core: tool results
        Core->>Events: emit(TOOL_COMPLETED)
    end

    Core->>Events: emit(GENERATION_COMPLETED)
    Core->>App: GenerateResponse

Loading

Note: in the Python API, execute_tools defaults to False (pass-through). Tool calls are returned in GenerateResponse.tool_calls for your host/runtime to execute. execute_tools=True exists for simple demos but is deprecated for most production use cases. The optional HTTP gateway server runs in pass-through mode.

5. Tool System Architecture

The tool system provides universal tool-call detection (and optional local execution) across all providers:

graph TD
    A[LLM Response] --> B{Has Tool Calls?}
    B -->|No| C[Return Response]
    B -->|Yes| D[Parse Tool Calls]
    D --> E[Event: TOOL_STARTED]
    E --> F{Event Prevented?}
    F -->|Yes| G[Skip Tool Execution]
    F -->|No| H[Execute Tools]
    H --> I[Collect Results]
    I --> J[Event: TOOL_COMPLETED]
    J --> K[Append Results to Response]
    K --> C

    style D fill:#ffeb3b
    style H fill:#4caf50
    style E fill:#ff9800

Loading

Tool Execution Flow

1. Tool Detection: Parse tool calls from LLM response
2. Event Emission: Emit TOOL_STARTED (preventable)
3. Optional local execution (deprecated): execute tools inside AbstractCore when execute_tools=True (providers never execute arbitrary local tools)
4. Result Collection: Gather results and error information
5. Event Emission: Emit TOOL_COMPLETED with results
6. Response Integration: Append tool results to original response

Provider-Specific Tool Handling with Tag Rewriting

graph LR
    A[Tool Definition] --> B{Provider Type}
    B --> C[OpenAI: Native JSON]
    B --> D[Anthropic: Native XML]
    B --> E[Ollama: Architecture-specific]
    B --> F[Others: Prompted Format]

    C --> G[LLM Generation]
    D --> G
    E --> G
    F --> G

    G --> H[Tool Call Tag Rewriter]
    H --> I[Target Format Conversion]
    I --> J[Universal Tool Parser]
    J --> K[Local Tool Execution]

    style A fill:#e1f5fe
    style H fill:#ff9800
    style I fill:#9c27b0
    style K fill:#4caf50

Loading

Tool Call Tag Rewriting System

AbstractCore includes a sophisticated tag rewriting system that enables compatibility with any agentic CLI:

Rewriting Pipeline:

graph TD
    A[Raw LLM Response] --> B[Pattern Detection]
    B --> C{Tag Format Needed?}
    C -->|No| D[Default Qwen3 Format]
    C -->|Yes| E[Target Format Conversion]

    E --> F{Format Type}
    F -->|Predefined| G[llama3, xml, gemma, etc.]
    F -->|Custom| H[User-defined Tags]

    G --> I[Rewritten Tool Call]
    H --> I
    D --> I

    I --> J[Tool Execution]

    style B fill:#2196f3
    style E fill:#ff9800
    style I fill:#4caf50

Loading

Supported Formats:

• Default (Qwen3): <|tool_call|>...JSON...</|tool_call|> - Compatible with Codex CLI
• LLaMA3: <function_call>...JSON...</function_call> - Compatible with Crush CLI
• XML: <tool_call>...JSON...</tool_call> - Compatible with Gemini CLI
• Gemma: tool_code...JSON... - Compatible with Gemma models
• Custom: Any user-defined format (e.g., [TOOL]...JSON...[/TOOL])

Real-Time Integration:

• Streaming Compatible: Works seamlessly with unified streaming architecture
• Zero Latency: No additional processing delays
• Universal Detection: Automatically detects source format from any model
• Graceful Fallback: Returns original content if rewriting fails

6. Retry and Reliability System

Production-grade error handling with multiple layers:

graph TD
    A[LLM Request] --> B[Retry Manager]
    B --> C{Error Type}
    C -->|Rate Limit| D[Exponential Backoff]
    C -->|Network Error| D
    C -->|Timeout| D
    C -->|Auth Error| E[Fail Fast]
    C -->|Invalid Request| E

    D --> F{Max Attempts?}
    F -->|No| G[Wait + Jitter]
    G --> H[Retry Request]
    H --> B
    F -->|Yes| I[Circuit Breaker]

    I --> J{Failure Threshold?}
    J -->|No| K[Return Error]
    J -->|Yes| L[Open Circuit]
    L --> M[Fail Fast for Duration]

    style D fill:#ff9800
    style I fill:#f44336
    style L fill:#d32f2f

Loading

Retry Configuration

from abstractcore import create_llm
from abstractcore.core.retry import RetryConfig

config = RetryConfig(
    max_attempts=3,           # Try up to 3 times
    initial_delay=1.0,        # Start with 1 second delay
    max_delay=60.0,           # Cap at 1 minute
    use_jitter=True,          # Add randomness
    failure_threshold=5,      # Circuit breaker after 5 failures
    recovery_timeout=60.0     # Test recovery after 1 minute
)

llm = create_llm("openai", model="gpt-4o-mini", retry_config=config)

7. Event System

Observability hooks through events:

graph TD
    A[LLM Operation] --> B[Event Emission]
    B --> C[Global Event Bus]
    C --> D[Event Listeners]

    D --> E[Monitoring]
    D --> F[Logging]
    D --> G[Cost Tracking]
    D --> H[Tool Control]
    D --> I[Custom Logic]

    E --> J[Metrics Dashboard]
    F --> K[Log Files]
    G --> L[Cost Alerts]
    H --> M[Security Gates]
    I --> N[Business Logic]

    style B fill:#9c27b0
    style C fill:#673ab7
    style H fill:#f44336

Loading

Event Types and Use Cases

from abstractcore.events import EventType, on_global

# Cost monitoring (best-effort estimate; based on token usage)
def monitor_costs(event):
    if event.type != EventType.GENERATION_COMPLETED:
        return
    cost = event.data.get("cost_usd")
    if isinstance(cost, (int, float)) and cost > 0.10:
        alert(f"High estimated cost: ${cost:.2f}")

# Tool monitoring
def log_tools(event):
    if event.type == EventType.TOOL_COMPLETED:
        log(f"Tool completed: {event.data.get('tool_name')}")

# Performance tracking
def track_performance(event):
    if event.type != EventType.GENERATION_COMPLETED:
        return
    duration_ms = event.data.get("duration_ms")
    if isinstance(duration_ms, (int, float)) and duration_ms > 10_000:
        log(f"Slow request: {float(duration_ms):.0f}ms")

on_global(EventType.GENERATION_COMPLETED, monitor_costs)
on_global(EventType.TOOL_COMPLETED, log_tools)
on_global(EventType.GENERATION_COMPLETED, track_performance)

8. Structured Output System with Streaming Integration

Type-safe responses with automatic validation, retry, and unified streaming:

graph TD
    A[LLM Generate] --> B{Streaming Mode?}
    B -->|Yes| C[Unified Streaming Processor]
    B -->|No| D[Standard JSON Parsing]

    C --> E[Incremental Tool Detector]
    E --> F[Real-time Chunk Processing]
    F --> G[Tool Call Detection]
    G --> H[Mid-Stream Tool Execution]

    D --> I[Parse JSON]
    I --> J{Valid JSON?}
    J -->|No| K[Retry with Error Feedback]
    J -->|Yes| L[Pydantic Validation]

    L --> M{Valid Model?}
    M -->|No| K
    M -->|Yes| N[Return Typed Object]

    K --> O{Max Retries?}
    O -->|No| A
    O -->|Yes| P[Raise ValidationError]

    style C fill:#4caf50
    style E fill:#2196f3
    style F fill:#ff9800
    style G fill:#9c27b0
    style K fill:#f44336

Loading

Unified Streaming Architecture

AbstractCore’s streaming system provides character-by-character streaming with incremental tool detection and optional tool-call syntax rewriting.

Architecture Components:

graph TD
    A[Provider chunks] --> B[UnifiedStreamProcessor]
    B --> HS["IncrementalHarmonySplitter<br/>(raw Harmony transcripts)"]
    HS -->|"final / commentary"| C
    HS -->|"analysis"| R["metadata.reasoning_delta"]
    HS -->|"to=functions.NAME"| TC
    B --> C[IncrementalToolDetector]
    C -->|"text"| D["Tag Rewriter<br/>(when tool_call_tags is set)"]
    C -->|"complete tool call"| TC["chunk.tool_calls"]
    C -->|"unclosed or unparsable call"| U["metadata.unparsed_tool_call<br/>on the final chunk"]
    D --> F[chunk.content]
    B --> T["Terminal chunk (last):<br/>finish_reason, usage,<br/>prompt_cache on MLX"]

    style B fill:#4caf50
    style C fill:#2196f3
    style D fill:#ff9800
    style HS fill:#9c27b0

Loading

Key Features:

1. Unified Streaming Strategy

   • Single consistent approach across all providers
   • Best-effort time-to-first-token (TTFT) telemetry for debugging
   • Minimal buffering (incremental parsing)

2. Incremental Tool Detection

   • Real-time tool call detection during streaming
   • Emits chunk.tool_calls as soon as a full tool call is detected
   • Handles partial tool calls across chunk boundaries

3. Character-by-Character Streaming

   • Handles micro-chunking from providers (very small deltas)
   • Intelligent buffering for partial tool calls
   • Robust parsing with auto-repair for malformed JSON

4. Stream end parity

   • The last chunk carries the call's finish_reason and usage; on MLX it also carries metadata["prompt_cache"] when the call has a prompt_cache_key, the same record a non-streamed call returns
   • A tool call that never closes, or that no parser accepts, is reported as metadata["unparsed_tool_call"] instead of appearing in the text
   • A ```json block is a tool call only when it names a tool offered for the call
   • Raw Harmony (GPT-OSS) transcripts are split by channel as they arrive; framing tokens never reach the text
   • See Tool Calling — Tool calls while streaming

5. Tool Call Tag Rewriting Integration

   • Real-time format conversion during streaming
   • Support for multiple formats (Qwen3, LLaMA3, Gemma, XML, custom)
   • Designed to avoid large buffering while keeping tool calls structured

Streaming with Tag Rewriting Example:

from abstractcore import create_llm, tool

@tool
def analyze_code(code: str) -> str:
    """Return a small, deterministic analysis."""
    return f"chars={len(code)}"

llm = create_llm("ollama", model="qwen3:4b-instruct")  # requires Ollama running (default: http://localhost:11434)
for chunk in llm.generate(
    "Write a Python function, then call analyze_code on it.",
    stream=True,
    tools=[analyze_code],
    tool_call_tags="llama3",  # Emit <function_call>...</function_call> style tags
):
    print(chunk.content or "", end="", flush=True)
    if chunk.tool_calls:
        print(f"\nTool calls: {chunk.tool_calls}")

# Output format: <function_call>{"name": "analyze_code"}...</function_call>

Implementation pointers (source of truth):

• Unified streaming + tool detection: abstractcore/providers/streaming.py
• Streaming wrapper + TTFT metadata: abstractcore/providers/base.py

Automatic Error Feedback

When validation fails, AbstractCore provides detailed feedback to the LLM:

# If LLM returns invalid data, AbstractCore automatically retries with:
"""
IMPORTANT: Your previous response had validation errors:
• Field 'age': Age must be positive (got -25)
• Field 'email': Invalid email format

Please correct these errors and provide valid JSON.
"""

9. Session Management

Simple conversation memory without complexity:

graph LR
    A[BasicSession] --> B[Message History]
    A --> C[System Prompt]
    A --> D[Provider Reference]

    B --> E["generate()"]
    C --> E
    D --> E

    E --> F[Add to History]
    F --> G[Return Response]

    A --> H["save() / load()"]
    H --> I[JSON Persistence]

    style A fill:#2196f3
    style B fill:#4caf50

Loading

10. Server Architecture (Optional Component)

The AbstractCore server provides OpenAI-compatible HTTP endpoints built on top of the core library:

graph TD
    A[HTTP Client] --> B[FastAPI Server]
    W[Browser] --> CON["/console web console"]
    CON --> B
    B --> C{Endpoint Router}

    C --> D["/v1/chat/completions, /v1/responses"]
    C --> E["/v1/embeddings"]
    C --> F["/v1/models, /providers"]
    C --> Img["/v1/images/*, /v1/audio/* (optional)"]
    C --> Cache["/acore/prompt_cache/*, /acore/blocs/*"]
    C --> Host["/acore/host, /acore/models/*, /acore/engines/*, /acore/jobs/*"]

    D --> H[Request Validation]
    E --> H
    Img --> H
    F --> I[Provider Discovery]
    D -->|"response_format"| SO["Structured-output handler<br/>(schema validation + retries)"]
    SO --> J

    H --> J[AbstractCore Library]
    I --> J
    Cache --> J
    Host --> LM[Local model management]

    J --> K[Provider Interface]
    K --> L[LLM Providers]

    style B fill:#4caf50
    style J fill:#e1f5fe
    style K fill:#f3e5f5

Loading

Architecture Layers:

1. HTTP Layer: FastAPI-based REST API with request validation
2. Translation Layer: Converts HTTP requests to AbstractCore library calls
3. Core Layer: Uses the full AbstractCore provider system
4. Response Layer: Transforms responses to OpenAI-compatible format

Key Capabilities:

• OpenAI Compatibility: Drop-in replacement for OpenAI API clients
• Universal Provider Access: Single API for all providers (OpenAI, Anthropic, Ollama, etc.)
• Format Conversion: Automatic tool call format conversion for agentic CLIs
• Structured Outputs: OpenAI response_format (json_object, json_schema) on /v1/chat/completions for every provider; the answer goes through the structured-output handler and is validated against the caller's schema (see Server)
• Streaming Support: Server-sent events for real-time responses
• Model Discovery: Dynamic model listing across all providers
• Embedding Support: Multi-provider embedding generation (remote OpenAI-compatible providers plus local backends)
• Optional Vision Endpoints: OpenAI-compatible /v1/images/generations, /v1/images/edits, /v1/videos/generations, and /v1/videos/edits can proxy to an upstream media server without local vision runtimes; /v1/vision/* local model control remains delegated to abstractvision when installed and configured. Deep provider catalog discovery is exposed separately at /v1/vision/providers/ and /v1/vision/models, and long image/video runs can use /v1/vision/jobs/images/* or /v1/vision/jobs/videos/* progress polling.
• Optional Audio Endpoints: OpenAI-compatible /v1/audio/transcriptions and /v1/audio/speech delegated to capability plugins (typically abstractvoice). Voice/profile and TTS model discovery are exposed at /v1/audio/voices and /v1/audio/speech/models.
• Prompt Cache Control Plane: /acore/prompt_cache/* endpoints for cache stats/set/update/fork/clear/key_meta on warm gateway runtimes or an upstream abstractcore.endpoint (best-effort).
• Runtime + Memory Visibility: /acore/models/* runtime load/list/unload plus lock/unlock model locks (a locked runtime refuses unloading without force), GET /acore/models/context_estimate context-fit estimates, and GET /acore/memory host memory snapshot. Loaded listings merge the host's local model-server sweep and stamp text and sweep records with registry-declared modalities, and every record with the observing machine's host_id/host_name — see Memory and Model Residency.

Request Flow Example:

sequenceDiagram
    participant Client
    participant Server as FastAPI Server
    participant Core as AbstractCore
    participant Provider as LLM Provider
    
    Client->>Server: POST /v1/chat/completions
    Server->>Server: Validate Request
    Server->>Core: create_llm(provider, model)
    Server->>Core: llm.generate(messages, tools)
    Core->>Provider: API call with retry logic
    Provider->>Core: Response
    Core->>Server: GenerateResponse (tool calls are returned, not executed)
    Server->>Server: Convert to OpenAI format
    Server->>Client: HTTP Response (streaming or complete)

Loading

Server Features:

• Automatic Retry: Built-in retry logic from core library
• Event System: Full observability through events
• Debug Logging: Comprehensive request/response logging
• Health Checks: /health endpoint for monitoring
• Interactive Docs: Auto-generated Swagger UI at /docs
• Multi-Worker Support: Production deployment with multiple workers

11. Local Model Management

The CLI (abstractcore host|models|engines ...), the server's /acore/host/profile, /acore/models/*, /acore/engines/* and /acore/jobs/* routes, the web console at /console and the terminal console (abstractcore-console, which drives the CLI) all call the same Python modules, so every surface returns the same JSON payloads. AbstractGateway re-exposes those payloads.

graph TD
    CLI["abstractcore host / models / engines"] --> CORE
    TUI["abstractcore-console (terminal)"] --> CLI
    WEB["/console (browser)"] --> HTTP
    HTTP["/acore/host/profile, /acore/models/*, /acore/engines/*, /acore/jobs/*"] --> CORE

    subgraph CORE[Shared modules]
        HP["utils.host_profile: OS, accelerator, memory ceiling, free disk"]
        FIT["utils.model_fit: fit verdict per artifact"]
        CAT["config.model_catalog: curated seed, recommended_text_model()"]
        MAT["config.model_materializer: installed models, download, delete, MTP companions"]
        ENG["config.engines: engine detection and install plans"]
        JOBS["config.host_jobs: host_job_v1 registry, progress, cancel"]
    end

    CAT --> HP
    CAT --> FIT
    CAT --> MAT
    CAT --> REG["assets: model_downloads_catalog.json, model_capabilities.json"]
    JOBS --> MAT
    JOBS --> ENG
    MAT --> STORES["Engine stores: Hugging Face cache, LM Studio models folder, Ollama"]
    JOBS --> PERSIST["~/.abstractcore/config/jobs/: snapshots and events.jsonl"]

Loading

• Reads never download. The catalog, the installed-model listing and provider loads read local evidence; only an explicit download (a CLI verb or a job) reaches the network. See Centralized Config.
• Long work is a job. Downloads, deletes and engine installs run as host_job_v1 jobs with progress, a plain end reason and cancellation. Hugging Face transfers run in a child process that a cancel stops at once.
• One recommendation. model_catalog.recommended_text_model() picks the recommended text model for this machine on every surface.

See Local Models, Local Engines, Web Console and Terminal Console.

12. Email

abstractcore.comms.email holds one mailbox per install. The CLI (abstractcore email ...), the /acore/email routes used by the web console, the terminal console (which drives the CLI) and the email tools all go through EmailAccountStore, so every surface applies the same policy, limits and TLS rules.

graph TD
    CLI["abstractcore email ..."] --> STORE
    TUI["abstractcore-console (terminal)"] --> CLI
    WEB["/console Email tab"] --> HTTP["/acore/email/* routes"]
    HTTP --> STORE
    HTTP --> DISC
    TOOLS["email tools (Agent email tools on)"] --> CTX

    subgraph EMAIL[abstractcore.comms.email]
        STORE["store.EmailAccountStore: account, connect, folder, email address"]
        DISC["discovery: discover_servers, server_defaults"]
        VAULT["vault.SecretVault: AES-256-GCM secret.enc, key file secrets/sealing.key (no keychain)"]
        CTX["context.EmailContext: guarded_send"]
        POL["policy: allowlist / denylist"]
        LIM["limits: per hour / per day, sends.json"]
        CLIENT["client.EmailClient: IMAP read-only, SMTP, verified TLS"]
    end

    CLI --> DISC
    STORE --> VAULT
    STORE --> CTX
    CTX --> POL
    CTX --> LIM
    CTX --> CLIENT
    CLIENT --> SERVERS["IMAP and SMTP servers"]

Loading

Discovery only reads public HTTPS and DNS answers and returns server settings; it never stores anything. See Email.

Architecture Benefits

1. Provider Agnostic

• Same code works everywhere: Switch providers by changing one line
• No vendor lock-in: Easy migration between cloud and local providers
• Consistent semantics: tools, streaming, and structured output follow the same API surface (provider/model differences still apply)

2. Production Ready

• Automatic reliability: Built-in retry logic and circuit breakers
• Comprehensive observability: Events for every operation
• Error handling: Proper error classification and handling

3. Extensible

• Event system: Hook into any operation
• Tool system: Add new tools easily
• Provider system: Add new providers with minimal code

4. Performance Optimized

• Lazy loading: Providers loaded only when needed
• Connection pooling: Reuse HTTP connections
• Efficient parsing: Optimized JSON and tool parsing

Extension Points

AbstractCore is designed to be extended:

Adding a New Provider

from abstractcore.providers.base import BaseProvider

class MyProvider(BaseProvider):
    def generate(self, prompt: str, **kwargs) -> GenerateResponse:
        # Implement provider-specific logic
        return GenerateResponse(content="...")

    def get_capabilities(self) -> List[str]:
        return ["text_generation", "streaming"]

Adding Tools

from abstractcore import tool

@tool
def my_custom_tool(param: str) -> str:
    """Custom tool that does something useful."""
    return f"Processed: {param}"

Performance Characteristics

AbstractCore’s overhead is usually small compared to model inference and network latency. If performance matters, benchmark on your target provider/model/hardware.

Common levers:

• Provider choice and base URL latency
• Concurrency (async + connection pooling)
• Streaming vs non-streaming
• Structured output (schema size, retry behavior)
• Tool execution strategy (pass-through vs host execution)

Security Considerations

1. Tool Execution Safety

• Local execution (optional): tool execution is local (never executed by the provider); by default tool calls are returned for your host/runtime to execute
• Event prevention: Stop dangerous tools before execution
• Input validation: Validate tool parameters
• Command sandbox: execute_command and shell_exec run inside the OS sandbox built from the run's workspace set (abstractcore.tools.sandbox: macOS sandbox-exec, Linux bubblewrap or Landlock). The most specific workspace wins on every platform: an allowed folder inside a refused one is reachable (bubblewrap binds it after masking the refused parent), a refused folder inside an allowed one refuses its subtree. See Tool Calling (Command sandbox).
• Browser probe scope: browser_probe serves a local page from a private loopback origin that answers only for files the run's sandbox can read; file:// and out-of-scope loads are refused and listed in its report.

2. API Key Management

• Environment variables: Secure key storage
• Avoid logging: treat logs as sensitive; do not log secrets (AbstractCore tries to avoid printing keys in logs)
• Provider isolation: Keys scoped to specific providers

3. Data Privacy

• Local options: Support for local providers (Ollama, MLX)
• No persistent storage by default: conversation state lives in memory (for example BasicSession) unless you explicitly save it or enable tracing/logging
• Transparent processing: All operations are observable through events

Testing Strategy

The repo uses a mix of unit tests and integration tests. Some tests are provider-/network-/hardware-dependent and are opt-in.

Quick pointers:

• Run: pytest -q
• Vision tests: tests/README_VISION_TESTING.md
• Seed tests: tests/README_SEED_TESTING.md
• Streaming/tool parsing tests: tests/streaming/ and tests/test_agentic_cli_compatibility.py
• Server/endpoint tests: tests/server/ and tests/test_abstractendpoint_singleton_provider.py

Integration with AbstractFramework

AbstractCore is a core package in the AbstractFramework ecosystem:

• AbstractFramework (umbrella): https://github.com/lpalbou/AbstractFramework
• AbstractCore (this repo): https://github.com/lpalbou/AbstractCore
• AbstractRuntime: https://github.com/lpalbou/abstractruntime

In this ecosystem, AbstractCore focuses on LLM I/O + provider abstraction, while AbstractRuntime focuses on durable execution (effects/tools/workflows/state). AbstractCore remains usable standalone; when you need durability/policy/sandboxing around tools, plug it into a runtime (for example AbstractRuntime).

graph TD
    subgraph "UI Layer (peers)"
        A[AbstractCode<br/>Terminal CLI]
        B[AbstractFlow Visual Editor<br/>React + ReactFlow]
    end

    A -.->|optional| F[AbstractFlow Engine]
    B --> F

    F --> C[AbstractAgent]
    A --> C
    C --> D[AbstractRuntime]
    D --> E[AbstractCore]
    E --> G[LLM Providers]

    style E fill:#e1f5fe
    style A fill:#fff3e0
    style B fill:#fff3e0
    style F fill:#f3e5f5
    style C fill:#f3e5f5
    style D fill:#f3e5f5

Loading

Framework Layers

• UI Layer (peers):
  • AbstractCode: Terminal CLI for interactive sessions
  • AbstractFlow Visual Editor: Web-based diagram editor (React + ReactFlow + FastAPI)
• AbstractFlow: Multi-agent orchestration engine + visual editor
• AbstractAgent: Agent patterns (ReactAgent, CodeActAgent) with durable execution
• AbstractRuntime: Effect system, workflows, state persistence

AbstractCode can optionally use AbstractFlow for running flows. AbstractFlow includes its own visual editor for designing workflows.

Summary

AbstractCore's architecture prioritizes:

1. Reliability - Production-grade error handling and retry logic
2. Simplicity - Clean APIs that are easy to understand and use
3. Universality - Same interface and features across all providers
4. Extensibility - Clear extension points for advanced features
5. Observability - Comprehensive events for monitoring and control
6. Flexibility - Deploy as Python library or OpenAI-compatible HTTP server

The result is a foundation that works reliably in production while remaining simple enough to learn quickly and flexible enough to build advanced applications on top of.