feat(x): AI image generation - #864
Open
prakhar1605 wants to merge 9 commits into
Open
Conversation
…del suggestions Generalizes generate-image beyond OpenRouter BYOK: resolveImageBackend() picks the first configured provider among openrouter/google/openai/ ollama/openai-compatible with per-flavor default models (Ollama runs via its OpenAI-compatible /v1/images/generations endpoint). Adds fail-fast runtime validation for aspectRatio, fixes the schema's example model ids (the old x-ai/grok-imagine example does not exist on OpenRouter and was copied verbatim by the chat model, causing avoidable 404s), and appends did-you-mean suggestions to OpenRouter model-404s from the public output_modalities=image catalog (3s timeout, fails silent to the unchanged error).
- Ollama default model id is x/z-image-turbo (library id was wrong; pull hint fixed too) - aspectRatio 'auto' is omitted instead of forwarded (Google rejects the literal) - provider warnings surfaced on the tool result; OpenAI gets ratio->size mapping so a 16:9 request no longer silently returns a square - ctx.signal propagated as abortSignal; abort rethrows so stopping a turn cancels the billed request instead of falling back - extension from image.mediaType, bytes written from uint8Array - resolveImageBackend hoisted to one call; random suffix on filenames - neutral comments, English-only tool description - renderer: 'Generating image' tool row label with prompt summary
The merge from main brought in file-path-card.test.tsx, which assumed image paths render as a routing card. On this branch images render as a ChatGPT-style inline picture that expands into the lightbox, so the test found no button role and failed in CI. Cover the inline behavior and the unreadable-image fallback to the routing card instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds real AI image generation via a new
generate-imagebuiltin tool, plus ChatGPT-style inline rendering of image file references in chat. Works today for BYOK users through any configured provider (OpenRouter, Google, OpenAI, Ollama, or an OpenAI-compatible server); the signed-in Rowboat-gateway path is fully implemented client-side and activates automatically once the backend supports it (see Gateway status). Users can name any OpenRouter image model in their prompt ("use grok imagine") to override the default on their own key; otherwise a fixed default runs.Why
The app has no image generation — asked for an image, agents fall back to drawing with Python/PIL or hand-writing SVG (reproducible pre-branch). Client root cause is threefold: the model picker filters to
tool_callmodels so image models can't be selected; requests never send OpenRouter'smodalities: ["image","text"]; and the stream bridge has no handler forfileparts, so image output would be silently dropped.Rather than making the chat model image-capable (new persisted part types, bridge changes — and image models can't tool-call, which breaks the agent loop), this follows the industry pattern (Open WebUI / OpenClaw / LibreChat) and our own web platform's Gemini tool: a builtin tool with an internally configured image model, decoupled from the chat model.
What changed
packages/core—runtime/tools/domains/image.ts(new), catalog + key-order test:generate-image(permissionnone):prompt, optionalfilename/aspectRatio/model.web-search: signed-in → gatewayimageModel(...); gateway failure → BYOK OpenRouter fallback with anotecarrying the gateway error; signed-out → BYOK directly. Both paths use AI SDKgenerateImage(); the gateway path works unchanged since it'screateOpenRouteratAPI_URL/v1/llm.google/gemini-2.5-flash-imageon both paths (separate consts; the gateway one is a placeholder for your pick). The per-callmodeloverride runs only on the user's own key — validated (vendor/model shape) before any API call; a signed-in override skips the gateway so Rowboat billing stays founder-controlled.<WorkDir>/generated_images/; result returns path/provider/model; the tool instructs the model to present the path as afilepathblock (renders via existing file cards).apps/renderer—file-path-card.tsx,lib/file-utils.ts(+ test):No changes to the stream bridge, turn schemas, persistence, or
packages/shared.Gateway status — backend action needed
Signed-in evidence:
imageModel("google/gemini-2.5-flash-image")against/v1/llm→ 404 Not Found; the tool fell back to the user's key and recordednote: "Rowboat gateway image call failed (Not Found); used your OpenRouter key instead."The client needs zero further changes once the backend exposes image models and passesmodalitiesthrough.Testing
tsc --noEmitclean (core + renderer); catalog key-order + permission tests 9/9; new file-utils test.x-ai/grok-imagine-image-qualityon the user's key, gateway skipped) — a wrong model id returns a readable 404 naming the id, which the chat model used to self-correct and retry; 402 and no-provider errors verified; non-image cards unchanged; plain questions don't trigger the tool.Limitations
Prompt-driven override only (no persistent user default yet); no image editing / image-to-image yet; no retention policy for
generated_images/; BYOK per-image cost isn't surfaced in-app.Questions
modalitiespass-through on/v1/llm, and how should image usage be billed/attributed?google/gemini-2.5-flash-image, your call.