Skip to content

faq.md

Latest commit

 

History

History
322 lines (212 loc) · 16.1 KB

faq.md

File metadata and controls

322 lines (212 loc) · 16.1 KB
Raw

FAQ

What do I get with pip install abstractcore?

The light install. It runs everything through remote inferencers and servers: OpenAI, Anthropic, OpenRouter, Portkey, any OpenAI-compatible /v1 endpoint, LM Studio, Ollama and a vLLM server all work without another install step. It also includes the built-in tools, media inputs (images, PDFs, Office documents), token counting, Glyph compression, the HTTP server, and the voice, vision, music and 3D capability plugins with their remote backends. It includes no local engine (no torch, transformers, MLX or vLLM). See Installation.

Which install do I need for my provider?

  • OpenAI, Anthropic, OpenRouter, Portkey, Ollama, LM Studio, a vLLM server, any OpenAI-compatible /v1 endpoint: pip install abstractcore.
  • MLX, or HuggingFace/GGUF models on a Mac: pip install "abstractcore[apple]".
  • A vLLM engine on this machine, or HuggingFace/GGUF models on an NVIDIA / AMD machine: pip install "abstractcore[gpu]".

There are only these three settings, and apple and gpu each include everything in light.

Why did my install pull torch / take a long time?

You installed abstractcore[apple] or abstractcore[gpu], which carry the local engines (torch, transformers, MLX or vLLM, local voice/image/music engines). The light install (pip install abstractcore) does not include torch or transformers.

What’s the difference between “provider” and “model”?

  • Provider: a backend adapter (openai, anthropic, ollama, lmstudio, …)
  • Model: a provider-specific model name (for example gpt-4o-mini or qwen3:4b-instruct-2507-q4_K_M)
from abstractcore import create_llm
llm = create_llm("openai", model="gpt-4o-mini")

How does AbstractCore relate to AbstractFramework / AbstractRuntime?

AbstractCore is one of the core packages in the AbstractFramework ecosystem:

AbstractCore is usable standalone. In the ecosystem, the common pattern is:

  • AbstractCore produces resp.content + resp.tool_calls
  • a runtime (for example AbstractRuntime) decides whether/how to execute tools (policy, sandboxing, retries, persistence)

See Architecture and Tool Calling.

How do I connect to a local server (Ollama / LMStudio / vLLM / llama.cpp / LocalAI)?

Use the matching provider and set base_url (or the provider’s base-url env var). We recommend open-source/local providers first; cloud and gateway providers are optional.

Examples:

from abstractcore import create_llm

llm = create_llm("ollama", model="qwen3:4b-instruct-2507-q4_K_M", base_url="http://localhost:11434")
llm = create_llm("lmstudio", model="qwen/qwen3-4b-2507", base_url="http://localhost:1234/v1")
llm = create_llm("vllm", model="Qwen/Qwen3-Coder-30B-A3B-Instruct", base_url="http://localhost:8000/v1")

For a generic OpenAI-compatible endpoint, use openai-compatible:

llm = create_llm("openai-compatible", model="my-model", base_url="http://localhost:1234/v1")

See Prerequisites for setup details and env var names.

Why do gateway providers return “unsupported parameter” errors (temperature/max_tokens)?

Gateways like Portkey and OpenRouter forward your payload to the routed backend model, and strict families (for example OpenAI reasoning models like gpt-5/o1) reject unsupported parameters.

In AbstractCore’s gateway providers:

  • Portkey uses PORTKEY_API_KEY and PORTKEY_CONFIG (config id) for routing.
  • Optional params (temperature, top_p, max_output_tokens) are only sent when you explicitly set them.
  • Reasoning families (gpt-5/o1) drop temperature/top_p and use max_completion_tokens instead of max_tokens.

If you still see errors, confirm:

  • You aren’t mixing routing modes (config vs virtual key vs provider-direct).
  • You’re not injecting parameters via Portkey config overrides that the backend rejects.

How do I set API keys and defaults?

You can use environment variables, or persist settings via the config CLI:

abstractcore --config
abstractcore --set-api-key openai sk-...
abstractcore --set-api-key anthropic sk-ant-...
abstractcore --status

Config is stored in ~/.abstractcore/config/abstractcore.json. See Centralized Config.

Can I use the HTTP server with only provider API keys?

Yes. You do not have to give a client the AbstractCore server auth token. If ABSTRACTCORE_AUTH_TOKEN is not configured, a client can bring its own upstream provider key, for example an Anthropic, OpenRouter, or Portkey key, by sending it as X-AbstractCore-Provider-API-Key.

That key is forwarded only to the provider requested by the model route, such as anthropic/..., openrouter/..., or portkey/.... It does not unlock other server-configured provider keys, and it does not grant access to providers the client did not supply credentials for.

If ABSTRACTCORE_AUTH_TOKEN is configured, Authorization is reserved for the AbstractCore server auth token. In that mode, use X-AbstractCore-Provider-API-Key only when you want to override the upstream provider key for a single request.

Provider keys in request bodies remain disabled. Select discovery endpoints accept an api_key query parameter for tooling/Swagger UI convenience, but headers remain preferred.

Why aren’t tools executed automatically?

By default, AbstractCore runs in pass-through mode (execute_tools=False): it returns tool calls in resp.tool_calls, and your host/runtime decides whether/how to execute them.

Automatic execution (execute_tools=True) exists but is deprecated for most use cases. See Tool Calling.

What’s the difference between web_search, skim_websearch, skim_url, and fetch_url?

These built-in web tools live in abstractcore.tools.common_tools and are part of the light install.

  • web_search: fuller DuckDuckGo result set (good when you want breadth or more options). It normalizes JSON-style numeric num_results values such as "5" before calling ddgs, and invalid values fail explicitly instead of silently changing the request.
  • skim_websearch: compact/filtered search results (good default for agents to keep prompts smaller). Defaults to 5 results, truncates long snippets, accepts string-like numeric num_results, and surfaces its compact cap when a caller asks for more than 15 results.
  • skim_url: fast URL triage (fetches only a prefix and extracts lightweight metadata + a short preview). It can also summarize RSS/Atom/XML feeds and PDF content when those assets are detectable from headers or content bytes. Defaults: max_bytes=200_000, max_preview_chars=1200, max_headings=8.
  • fetch_url: full fetch + parsing for HTML, JSON, XML/RSS/Atom, plain text, and PDFs. PDFs are extracted locally (pypdf, then pymupdf when installed) and never leave the machine by default, whatever API keys are configured. Sending a fetched PDF to the configured OpenAI-compatible LLM for extraction and a summary is an explicit opt-in: abstractcore --allow-remote-pdf-extraction (config key offline.allow_remote_pdf_extraction). Once enabled, native PDF calls try file_id first and then data_url when an OpenAI-compatible endpoint rejects uploaded PDF files. Results expose PDF backend provenance, including native transport. It downloads up to 10MB by default; use include_full_content=False for smaller outputs.

Recommended workflow: skim_websearch → skim_url → fetch_url (use include_full_content=False when you want a smaller fetch_url output).

How do I preserve tool-call markup in response.content for agentic CLIs?

Use tool-call syntax rewriting:

  • Python: pass tool_call_tags=... to generate() / agenerate()
  • Server: set agent_format in requests

See Tool Syntax Rewriting.

How do I get structured output (typed objects) instead of parsing JSON?

Pass a Pydantic model via response_model=...:

from pydantic import BaseModel
from abstractcore import create_llm

class Answer(BaseModel):
    title: str
    bullets: list[str]

llm = create_llm("openai", model="gpt-4o-mini")
result = llm.generate("Summarize HTTP/3 in 3 bullets.", response_model=Answer)

Over the OpenAI-compatible server, send response_format ({"type": "json_object"} or a json_schema) to POST /v1/chat/completions; the answer is validated against your schema on every provider. See Server: structured outputs.

See Structured Output.

Why does structured output retry or fail validation?

Structured output is validated against your schema. If validation fails, AbstractCore retries with feedback (up to the configured retry limit). Common fixes:

  • simplify schemas (fewer nested structures; fewer strict constraints)
  • tighten prompts (be explicit about allowed values and ranges)
  • increase timeouts for slow backends

See Structured Output and Troubleshooting.

How do I attach PDFs / Office docs / images?

Media handling is part of the light install. Pass media=[...] to generate() or use the media pipeline. See Media Handling.

How do I attach audio or video?

Audio and video attachments are supported via media=[...], but they are policy-driven by design:

  • Audio defaults to audio_policy="native_only" (fails loudly unless the model supports native audio input).
  • Video defaults to video_policy="auto" (native video when supported; otherwise sample frames and route through image/vision handling). Frame sampling requires ffmpeg/ffprobe.

Speech-to-text fallback for audio (audio_policy="speech_to_text" or "auto") runs through the AbstractVoice capability plugin (part of the light install).

You can set defaults via the config CLI:

abstractcore --set-audio-strategy auto
abstractcore --set-video-strategy auto
abstractcore --set-video-max-frames 6

See:

How do I do speech-to-text (STT) or text-to-speech (TTS)?

The light install includes the AbstractVoice capability plugin with its remote backends. Local voice engines come with abstractcore[apple] or abstractcore[gpu].

Then use the deterministic capability surfaces:

from abstractcore import create_llm

llm = create_llm("openai", model="gpt-4o-mini")  # provider/model is only for LLM calls; STT/TTS are deterministic
print(llm.capabilities.status())  # shows which capability backends are available/selected

wav_bytes = llm.voice.tts("Hello", format="wav")
text = llm.audio.transcribe("speech.wav")

If you run the optional HTTP server, you can also use OpenAI-compatible endpoints:

  • POST /v1/audio/transcriptions
  • POST /v1/audio/speech

See: Server and Capabilities.

How do I generate or edit images?

The light install includes the AbstractVision plugin, and AbstractCore Server can proxy OpenAI-compatible image endpoints without local vision runtimes. For local image and video generation (MLX-Gen, Diffusers, stable-diffusion.cpp), install the local engines:

pip install "abstractcore[apple]"   # Apple silicon
pip install "abstractcore[gpu]"     # NVIDIA / AMD

You can use generative vision through AbstractCore’s llm.vision.* capability plugin surface, or through AbstractCore Server’s optional endpoints:

  • POST /v1/images/generations
  • POST /v1/images/edits

Omit model with the server endpoints only when this server has a configured AbstractVision/OpenAI-compatible image default. Use model="diffusers/default" or model="diffusers/<huggingface-repo>" for explicit Diffusers routing, model="sdcpp/default" for configured stable-diffusion.cpp, or model="openai-compatible/<model>" with a configured image base URL for remote OpenAI-compatible endpoints. Local Diffusers is cache-only by default, so pre-download the model or set ABSTRACTCORE_VISION_ALLOW_DOWNLOAD=1 / ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD=1 when runtime downloads are intended.

See: Server, Capabilities, and abstractvision/docs/reference/abstractcore-integration.md (in the AbstractVision repo).

What are “glyphs” and what do they require?

Glyph visual-text compression is an optional feature for long documents. Its renderer and PDF extraction are part of the light install.

See Glyph Visual-Text Compression.

How do I use embeddings?

Remote embeddings (OpenAI, OpenRouter, Portkey, Ollama, LM Studio, OpenAI-compatible endpoints) work with the light install. Local embedding models (sentence-transformers) come with the local engines:

pip install "abstractcore[apple]"   # Apple silicon
pip install "abstractcore[gpu]"     # NVIDIA / AMD

Then import from the embeddings module:

from abstractcore.embeddings import EmbeddingManager

See Embeddings.

Do I need the HTTP server?

No. The server is optional and is mainly for:

  • exposing one OpenAI-compatible /v1 endpoint that can route to multiple providers/models
  • integrating with OpenAI-compatible clients and agentic CLIs

It is part of the light install. Run:

pip install abstractcore
abstractcore serve        # open the printed link for the web console

On your own machine the server creates its bearer token on first run (abstractcore serve --print-token prints it). See Server.

Where are logs and traces?

I’m getting HTTP timeouts. What should I change?

  • Per-provider: pass timeout=... to create_llm(...) (timeout=None means unlimited).
  • Process-wide default: set abstractcore --set-default-timeout 0 (0 = unlimited), or set a larger value.
  • Some CLI apps have their own --timeout flags; run --help for the exact behavior.

See Troubleshooting and Centralized Config.

HuggingFace won’t download models — why?

The HuggingFace provider respects AbstractCore’s offline-first settings. A load reads the local cache only, and a model that is not cached fails at once with download it first: abstractcore models download huggingface <repo>. Downloading it that way is the recommended fix. If you want loads themselves to fetch from the Hub, update ~/.abstractcore/config/abstractcore.json:

  • set "offline_first": false
  • set "force_local_files_only": false

Restart your Python process after editing the file (the configuration is loaded once per process). Neither setting touches HF_HUB_OFFLINE in the process. See Centralized Config.

Does native MLX MTP support vision and continuous batching?

Native image input works with MTP off or on for the documented Qwen3.8-27B and Flash-Next artifacts. MTP does not replace or bypass their vision encoder. The vision comparison uses swapped-image answers and actual draft/verify execution evidence.

With mlx_batching=True, compatible greedy target-only requests use continuous admission. MTP requests form fixed same-depth cohorts; later arrivals wait. Sampled requests run exclusively. For concurrent or staggered clients, compare speculation=False against MTP rather than assuming speculation is faster. See runtime policies.

Is AbstractCore a full agent/RAG framework?

AbstractCore focuses on provider abstraction + infrastructure (tools, structured output, media handling, tracing). It does not ship a full RAG pipeline or multi-step agent orchestration. See Capabilities.