Skip to content

feat(x): AI image generation - #864

Open
prakhar1605 wants to merge 9 commits into
mainfrom
feat/image-generation
Open

feat(x): AI image generation #864
prakhar1605 wants to merge 9 commits into
mainfrom
feat/image-generation

Conversation

@prakhar1605

@prakhar1605 prakhar1605 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds real AI image generation via a new generate-image builtin tool, plus ChatGPT-style inline rendering of image file references in chat. Works today for BYOK users through any configured provider (OpenRouter, Google, OpenAI, Ollama, or an OpenAI-compatible server); the signed-in Rowboat-gateway path is fully implemented client-side and activates automatically once the backend supports it (see Gateway status). Users can name any OpenRouter image model in their prompt ("use grok imagine") to override the default on their own key; otherwise a fixed default runs.

Why

The app has no image generation — asked for an image, agents fall back to drawing with Python/PIL or hand-writing SVG (reproducible pre-branch). Client root cause is threefold: the model picker filters to tool_call models so image models can't be selected; requests never send OpenRouter's modalities: ["image","text"]; and the stream bridge has no handler for file parts, so image output would be silently dropped.

Rather than making the chat model image-capable (new persisted part types, bridge changes — and image models can't tool-call, which breaks the agent loop), this follows the industry pattern (Open WebUI / OpenClaw / LibreChat) and our own web platform's Gemini tool: a builtin tool with an internally configured image model, decoupled from the chat model.

What changed

packages/coreruntime/tools/domains/image.ts (new), catalog + key-order test:

  • generate-image (permission none): prompt, optional filename / aspectRatio / model.
  • Provider selection mirrors web-search: signed-in → gateway imageModel(...); gateway failure → BYOK OpenRouter fallback with a note carrying the gateway error; signed-out → BYOK directly. Both paths use AI SDK generateImage(); the gateway path works unchanged since it's createOpenRouter at API_URL/v1/llm.
  • Default google/gemini-2.5-flash-image on both paths (separate consts; the gateway one is a placeholder for your pick). The per-call model override runs only on the user's own key — validated (vendor/model shape) before any API call; a signed-in override skips the gateway so Rowboat billing stays founder-controlled.
  • Output: PNG saved under <WorkDir>/generated_images/; result returns path/provider/model; the tool instructs the model to present the path as a filepath block (renders via existing file cards).
  • Registered at the end of the catalog so existing key order (cached prompt-prefix bytes) is byte-identical; golden test updated.

apps/rendererfile-path-card.tsx, lib/file-utils.ts (+ test):

  • Image filepath blocks render as just the image — no card/filename/Open button; rounded, click-to-open, hover-only open affordance. Reuses the existing image-viewer loading mechanism; load failure falls back to the plain card; non-image paths byte-identical. Benefits all image references in chat, not just generated ones.

No changes to the stream bridge, turn schemas, persistence, or packages/shared.

Gateway status — backend action needed

Signed-in evidence: imageModel("google/gemini-2.5-flash-image") against /v1/llm404 Not Found; the tool fell back to the user's key and recorded note: "Rowboat gateway image call failed (Not Found); used your OpenRouter key instead." The client needs zero further changes once the backend exposes image models and passes modalities through.

Testing

  • tsc --noEmit clean (core + renderer); catalog key-order + permission tests 9/9; new file-utils test.
  • Manual (macOS dev): BYOK generates, saves, renders inline; signed-in exercises gateway → 404 → silent fallback end-to-end; override verified live (x-ai/grok-imagine-image-quality on the user's key, gateway skipped) — a wrong model id returns a readable 404 naming the id, which the chat model used to self-correct and retry; 402 and no-provider errors verified; non-image cards unchanged; plain questions don't trigger the tool.
  • Not yet tested: Windows/Linux (@gagan / @harsh — save path + rendering check would be great), packaged build.

Limitations

Prompt-driven override only (no persistent user default yet); no image editing / image-to-image yet; no retention policy for generated_images/; BYOK per-image cost isn't surfaced in-app.

Questions

  1. Backend: expose image models + modalities pass-through on /v1/llm, and how should image usage be billed/attributed?
  2. Official gateway default model — placeholder is google/gemini-2.5-flash-image, your call.
  3. Follow-up PR: persistent default-model override in config — models.json or app settings?

prakhar1605 and others added 9 commits August 14, 2026 00:06
…del suggestions

Generalizes generate-image beyond OpenRouter BYOK: resolveImageBackend()
picks the first configured provider among openrouter/google/openai/
ollama/openai-compatible with per-flavor default models (Ollama runs via
its OpenAI-compatible /v1/images/generations endpoint). Adds fail-fast
runtime validation for aspectRatio, fixes the schema's example model ids
(the old x-ai/grok-imagine example does not exist on OpenRouter and was
copied verbatim by the chat model, causing avoidable 404s), and appends
did-you-mean suggestions to OpenRouter model-404s from the public
output_modalities=image catalog (3s timeout, fails silent to the
unchanged error).
- Ollama default model id is x/z-image-turbo (library id was wrong; pull hint fixed too)
- aspectRatio 'auto' is omitted instead of forwarded (Google rejects the literal)
- provider warnings surfaced on the tool result; OpenAI gets ratio->size mapping
  so a 16:9 request no longer silently returns a square
- ctx.signal propagated as abortSignal; abort rethrows so stopping a turn
  cancels the billed request instead of falling back
- extension from image.mediaType, bytes written from uint8Array
- resolveImageBackend hoisted to one call; random suffix on filenames
- neutral comments, English-only tool description
- renderer: 'Generating image' tool row label with prompt summary
The merge from main brought in file-path-card.test.tsx, which assumed
image paths render as a routing card. On this branch images render as a
ChatGPT-style inline picture that expands into the lightbox, so the test
found no button role and failed in CI. Cover the inline behavior and the
unreadable-image fallback to the routing card instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant