Add ComponentController tools - #2056
philippjfr wants to merge 3 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2056 +/- ##
==========================================
+ Coverage 72.48% 75.77% +3.28%
==========================================
Files 203 218 +15
Lines 35960 40665 +4705
==========================================
+ Hits 26066 30814 +4748
+ Misses 9894 9851 -43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ghostiee-11
left a comment
ghostiee-11
left a comment
There was a problem hiding this comment.
Ran the new suite locally, 41 passed. A few things I ran into while reading through it. Also lumen/tests/ai/test_components.py is added as an empty file, I think that one can just be dropped.
| @@ -0,0 +1,59 @@ | |||
| """ | |||
There was a problem hiding this comment.
parse_literal and fetch_url aren't exported from tools/__init__.py or used anywhere, and aren't mentioned in the description. Did this file get committed by accident?
|
|
||
| @wrap_logfire(span_name="Chat Invoke") | ||
| async def _chat_invoke(self, contents: list | str, user: str, instance: ChatInterface) -> Plan: | ||
| async def _chat_invoke(self, contents: list | str, user: str, instance: ChatInterface) -> None: |
There was a problem hiding this comment.
The _resolve_plan agents.get("ChatAgent") fix from the description isn't in the diff, planner.py:488 still indexes agents["ChatAgent"].
| # cannot be serialized back into the conversation on the next turn. | ||
| if plan is not None: | ||
| with plan.param.update(interface=self.interface): | ||
| await plan.execute() |
There was a problem hiding this comment.
UI._chat_invoke (ui.py:709) drops the plan the same way and ChatUI doesn't override it, so a plain ChatUI still plans and then does nothing. Fix both?
| represented in a JSON schema. | ||
| """ | ||
| if options is not None: | ||
| literal = Literal[tuple(list(options)[:MAX_OPTIONS])] |
There was a problem hiding this comment.
Past 50 options the Literal truncates the enum so the model can't pick the rest, even though _coerce_option would accept them. Falling back to str when it overflows would keep them reachable.
This branch has not been deployed
2 participants
Problem
lumen.ai can answer questions about data, but it cannot touch the application it lives in. A
dashboard that already has the right filters, the right axes and the right model parameters
still forces the user to find and move every widget by hand, and the assistant sitting next
to those widgets can only describe what the user should click.
There was also no way to point an assistant at an application at all. Agents and tools take
data sources, not widgets, so exposing "the eight controls in the sidebar" meant hand-writing
one
FunctionToolper widget, repeating each label, each set of options and each bound asprose in the tool description, and rewriting all of it whenever the layout changed.
Approach
Widgets already carry everything needed to describe themselves. A
param.Parameterknows itstype, its bounds, its allowed values and its
docstring, a widget knows itslabelanddescription, andparamknows how to serialize and deserialize a value. So the tools arederived rather than declared: hand over the components and the schemas fall out of them.
Changes
ComponentController(lumen/ai/tools/component_control.py)Takes a widget, a list, a dict keyed by the name the LLM should use, or an entire layout such
as a
panel_material_ui.Page, and produces:list_ui_components, an overview of every component with its current value,describe_ui_component, the full parameter list of one component withdocstrings,bounds and allowed values,
set_<key>per component, andclick_<key>instead, when a component's only settable parameter is aparam.Event.Pass it anywhere
llm_toolsare accepted (ExplorerUI(llm_tools=[controller])) or hand it toa
ComponentControlAgent.What gets exposed. Widgets expose
valuealone and take their meaning fromlabelanddescription. Any otherParameterizedobject exposes every parameter it declares itself,with the inherited
Parameterized/Layoutable/Viewablechrome filtered out, so a domainmodel can be handed over directly and its parameter docstrings become the tool documentation.
constantandreadonlyparameters produce no setter but are still reported, which is how amodel reads back the effect of a change.
descriptions,parametersandexcludecover thecases where the component does not describe itself well enough.
Values go through param.
_serializer/_deserializeruse a parameter's ownserialize/deserializewhen the class overrides the identity implementations onparam.Parameter, so dates, ranges and dataframes round-trip in the form the parameterexpects instead of whatever JSON the model guessed at. Writes are validated by
paramand arejected value comes back to the model as a message it can act on, along with the constraint
it violated. Selector constraints are rendered as "any of" for multi-valued parameters and
"one of" for single-valued ones, since a model that is told "one of" will not send a list.
Live layouts.
specsre-resolves on every access rather than at construction, so acontroller pointed at a
Pagepicks up components added or removed at runtime, and listingthe controller in
llm_toolsis enough for the tool set handed to the LLM to describe theapplication as it is now. Layout keys come from slugified labels, deduplicated with a numeric
suffix; explicit dict keys always win.
ComponentControlAgent(lumen/ai/agents/component_control.py)Wires a controller into a coordinator. Its
applieshook writes the current list of controlsonto the
purpose, because a coordinator routes on the purpose alone and "show only Gentoopenguins" is only distinguishable from a data query if the router can see that a species
filter exists. Its prompt covers three things beyond plain writes: relative changes ("a bit
stronger"), the fact that read-only values only reflect a change after the listing tool is
called again, and goal seeking, where the agent writes a candidate, re-reads the resulting
state and iterates instead of asking the user to try values themselves. For a "weakest
setting that still meets the target" request it is told to step back towards what the user
asked for while the target holds and to never leave the application in a state that fails it.
Two fixes found on the way
Coordinator._chat_invokeplanned a response and returned thePlanwithout running it. AUIsupplies its own_chat_invokeand so was unaffected, but a coordinator used on its own,which is exactly what a small embedded assistant is, produced a plan and then did nothing. It
now executes the plan and returns
None, since aPlanplaced in the message feed cannot beserialized back into the conversation on the next turn.
Planner._resolve_planlooked upagents["ChatAgent"]unconditionally to checknot_with,so any agent set without a
ChatAgentdied withKeyErrorbefore running a single step. Itnow uses
agents.getand only appends the summarize step when that agent exists.Demos
examples/ai/penguin_copilot.pywalks a wholePageand exposes 11 components as 13 tools,with the assistant in a docked
Draweron the right. "Colour by sex and plot flipper lengthagainst body mass" moves four widgets in one turn.
examples/ai/epidemic_copilot.pyis an SEIR outbreak simulator with disease presets, asaveable baseline, KPI cards and a hospital demand plot drawn relative to capacity. It hands
over the model, a read-only
Outcomeand the display controls, so the assistant can both actand observe. That is what makes "find the weakest lockdown that keeps hospital load under
100%" work: it searches down from an intervention strength of 0.7 to 0.495 and settles on a
97.5% peak load with zero overflow days. The demo pins the larger model, since the smaller
ones tend to stop after the first write rather than iterate.
Tests
41 tests in
lumen/tests/ai/test_component_control.py, covering spec discovery for widgetsand plain
Parameterizedobjects, walking a layout, re-resolution after the layout changes,key collisions, serializer round-trips, rejected writes, trigger detection, per-component
overrides and the generated summaries and schemas.
Both demos were verified three ways: rendered into a bokeh document, screenshotted through
Playwright against
panel servewith no console errors, and driven by real LLM runs thatchanged presets, population, plotted curves, the axis scale and the baseline button, and that
carried out the bounded search described above.
Existing suite: 89 passed in the planner and coordinator tests. The 8 failures in
lumen/tests/aiare pre-existing and unrelated (anasyncio.iscoroutinefunctiondeprecation raised as an error, two prompt-heading checks for unrelated in-progress agents,
param_to_pydantic, two xarray upload handlers, a csvresolve_dataand an edit callback).AI Disclosure
Written with assistance of Claude Opus 5