Recipe needed: how should an agent *look* at a visual cell output?

[shntnu]

Problem

The skill's core mandate is to trust a result only after looking at the output - but that assumes human eyes. A live kernel returns text reprs; an altair/vega chart's repr carries no visual information. An agent has no sanctioned way to actually see a rendered chart, so it improvises. This bites even inside a proper marimo-pair session, not just headless flows.

Current behavior

marimo-pair returns text/repr from the kernel. There is no documented path to get a cell's rendered output back to the agent as an image.

Workaround used in practice

Render the chart to PNG and Read the file - which is how an agent caught two real bugs a text repr would have hidden (combo labels overflowing their wells; viability plates silently collapsing out empty rows/columns so they no longer matched the sibling plates):

import altair as alt  # chart object `ch`
ch.save("/tmp/out.png", ppi=120)   # needs vl-convert-python
# then Read /tmp/out.png

Considerations

• Scope is a recipe, not a new _code_mode API. Likely home: reference/rich-representations.md.
• Dependency question: bless vl-convert-python for altair, or reuse an existing mo screenshot/save mechanism?
• Generality: matplotlib already rasterizes; altair needs the save step; interactive widgets do not rasterize at all - the recipe should state what is coverable and what is not.
• A native "capture this output as an image" in _code_mode stays a possible future, explicitly out of scope here.

[manzt]

There's no single answer because it somewhat depends on how "headed" your pairing environment is. That said, we could improve the recommended/discoverability of these things in the skill itself to give the agent more direction to happy path.

We added the ability to screenshot rendered cell outputs directly from a session with ctx.screenshot(target) (marimo-team/marimo#9232), which rasterizes the DOM to PNG so it works even for vega-embed, plotly, and interactive widgets that a text repr or ch.save() would miss.

You can also reach the session data directly through NotebookCell.output and .console_outputs (marimo-team/marimo#9653), and if the output is already an image mimetype like matplotlib you just read the bytes off it with no browser involved.

Beyond that it really comes down to the environment. If your headed environment is a browser then something like agent-browser works well (we lean on that a lot for testing marimo with marimo-pair). If you're in an IDE agent like VS Code's agent view, that has an integrated web browser that can capture the page as context and feed it back into your prompt. So the capability is supported, but the happiest path kind of depends on your setup.

[shntnu]

Please note: This was generated by an AI assistant and reviewed by me before posting.

Would you prefer I file this against marimo core (an opt-in flush-before-capture, or a wait_for_rerender / version-token barrier on screenshot), or keep it as a recipe caveat here?

---

Following up after testing the ctx.screenshot(target) path recommended above.

If an agent changes widget state and screenshots the affected output in the same code-mode block, the screenshot captures the old rendered output.

Shape of the repro:

import marimo._code_mode as cm

async with cm.get_context() as ctx:
    ctx.set_ui_value(dropdown, ["new value"])
    ctx.run_cell(output_cell_id)
    await ctx.screenshot(output_cell_id, save_to="out.png")

In my local repro on marimo 0.23.10, out.png was byte-for-byte identical to the screenshot taken before the widget value changed.

It's an ordering bug, confirmed against the 0.23.10 source (marimo/_code_mode/_context.py):

• set_ui_value(...) only appends to self._ui_updates, and run_cell(...) only does self._cells_to_run.add(...) - neither touches the kernel inside the block.
• Both flush to the kernel only in __aexit__ (_kernel.run(...) and _kernel.set_ui_element_value(...)).
• screenshot(...) awaits capture() immediately, driving headless Chromium against the current DOM.

So in a same-block capture nothing has been dispatched yet, the DOM is literally unchanged, and the PNG is byte-identical rather than a frame behind.

The fix: change state in one context, let it exit, then screenshot in the next. That's reliable here but timing-dependent - context exit flushes kernel-side, yet the kernel-to-frontend update has no rerender ack to wait on, so it's a heuristic, not a guaranteed barrier.