Wire structured AI edits and simplify providers

Author: KSemenenko

AGENTS.md

@@ -162,6 +162,7 @@ Rule format:
 - For task-scoped work, edit, stage, and commit only the files directly required for the requested change; do not widen the change set into unrelated user-owned or parallel worktree edits, do not touch changes owned by another agent, and if a blocker comes from that parallel work, wait briefly and re-check instead of patching around their in-flight fix.
 - Core AI infrastructure should stay minimal and predefined: prefer one factory that assembles known agents and known workflows, and avoid extra catalog or registry layers for skills, agents, or workflows unless a later task explicitly requires dynamic discovery.
 - All AI provider connections must stay on the Microsoft Agent Framework composition path used by PrompterOne agents and `ChatClientAgent`; do not bypass it with direct provider-specific SDK clients for production graph, assistant, or runtime AI behavior, because Agent Framework is the single configuration, provider selection, and orchestration boundary for every AI integration. Provider SDK objects may exist only as private adapters that feed the Agent Framework path; production calls must not execute through those SDK clients directly.
+- AI provider configuration must stay minimal and user-owned: do not ship hardcoded model/deployment presets, token limits, context-size fields, model-type fields, or hidden defaults in Settings or appsettings. A provider entry should capture only the provider kind, editable endpoint/base URL when needed, API key when needed, and user-added model or deployment names, with exactly one active provider at runtime unless a later task introduces explicit routing.
 - Core AI skill and article wiring should use official Microsoft Agent Framework context primitives such as `AgentSkillsProvider` and `AIContextProvider`; do not manually concatenate embedded skill markdown into an agent system prompt.
 - AI, agent context, script graph extraction, and assistant action-matching paths must not use ad-hoc language heuristics for meaning, intent, or similarity. Use an available LLM path for semantic understanding; if no LLM is available, only an explicit tokenizer/vector similarity fallback is allowed, including for action search by tokenizing action descriptions and the user query into comparable token vectors.
 - Script graph tokenizer/vector fallback must use the tokenizer/vector primitives shipped by `ManagedCode.MarkdownLd.Kb` when available; do not keep a parallel PrompterOne-owned tokenizer implementation beside the graph library.
@@ -176,6 +177,7 @@ Rule format:
 - The PrompterOne AI assistant belongs in global shell/header chrome, must be available on every routed screen through a spotlight-style surface and hotkey, and must not remain an editor-only action.
 - The AI spotlight is a command/search palette plus agent chat/execution surface, not a planning screen: it should suggest route-aware commands and navigation targets while typing, send the entered prompt to the agent on submit, show execution progress only as live process/status feedback, and use a macOS Spotlight-style overlay where the underlying route remains visible with only a very light dim and no dark blurred full-page backdrop.
 - AI spotlight must not fake agent execution: when a provider is configured, submitting a prompt must invoke a real Microsoft Agent Framework agent with the user prompt, route/editor context, and MCP-style tool catalog; when no provider is configured, the UI must clearly report that AI is unavailable instead of showing a pretend working state.
+- AI assistant responses must use structured output instead of free-form parsing: separate the visible chat reply from machine-readable document edit operations, and express document changes as exact range-based insert, replace, or delete operations that can be applied independently of the chat text.
 - The AI spotlight prompt input must receive focus immediately when the spotlight opens so users can start typing without an extra click.
 - Script graph views must prioritize a readable writer knowledge map over decorative circular placement or raw TPS/source dumps: highlight the main/root document node, group meaningful document structure, ideas, people, themes, story references, and relationships by graph value/kind, expose node and edge hover tooltips with source detail, provide zoom/fit/layout controls, avoid noisy duplicated node-list strips under the graph, and never present source lines, WPM/pace, timing, cue, archetype, or raw TPS attribute values as primary graph nodes or primary edge labels. Graph viewing must support both a full graph-only mode and a split source/graph mode with a user-draggable divider that resizes the zones. Graph extraction must be useful to scriptwriters at a high-level glance: it should show what the text is about, how blocks and ideas connect, which references or entities recur, and where the user can jump back into the source; regex, stop-word, capitalization, keyword, and hardcoded-domain semantic heuristics are strictly forbidden. The only non-LLM fallback is an explicit user-requested tokenizer/vector similarity mode based on TikToken-style token vectors and distance calculations.
 - Script graph labels, details, semantic scopes, tokenizer chunks, and knowledge-graph markdown input must use compiled TPS display text from the TPS SDK, matching the clean text shown in the prompter; raw TPS source may be used for editor ranges and metadata only, not as a post-hoc string-cleaned source for visible graph prose.

README.md

@@ -53,8 +53,8 @@ No PrompterOne backend. No desktop install. No account wall. Open the app, start
 - **Learn**: rehearse with ORP-aligned RSVP, context rails, phrase-aware timing, WPM controls, stepping, looping, and punctuation-safe word progression.
 - **Teleprompter**: read with persisted font and width controls, focal-line positioning, horizontal and vertical mirror toggles, orientation switching, browser fullscreen, segmented progress, and optional camera background.
 - **Onboarding**: walk the first-run flow with a localized tour that explains TPS, RSVP, the editor, Learn, Teleprompter, and Go Live, then reopen that tour later from Settings.
-- **Settings**: manage appearance, browser language, media permissions, camera and microphone setup, sync offsets, recording defaults, AI provider preferences, cloud snapshot targets, transport credentials, and onboarding restart from one routed screen.
-- **AI Spotlight**: when an AI provider is configured, run the global assistant as a real Microsoft Agent Framework agent with route context, editor text, selected ranges, graph summary, and PrompterOne MCP-style tools.
+- **Settings**: manage appearance, browser language, media permissions, camera and microphone setup, sync offsets, recording defaults, minimal active AI provider setup, cloud snapshot targets, transport credentials, and onboarding restart from one routed screen.
+- **AI Spotlight**: when an AI provider is configured, run the global assistant as a real Microsoft Agent Framework agent with route context, editor text, selected ranges, graph summary, PrompterOne MCP-style tools, and structured output that separates chat replies from exact document edits.
 - **Go Live**: run the browser-owned studio shell and save the composed program feed locally with decodable video and audio while remote transport and destination routing continue to expand.
 
 ## What You Get
@@ -196,7 +196,7 @@ Distribution targets such as YouTube, Twitch, and custom RTMP are capability-gat
 
 ### Settings
 
-Settings holds the operational state for the rest of the app: appearance, browser language, cloud snapshot targets, camera selection with preview, microphone setup with live meters, delay and sync offsets, output quality profiles, recording defaults, AI provider preferences, transport credentials, and onboarding restart. Theme changes and layout preferences persist, and appearance changes propagate across tabs instead of drifting out of sync.
+Settings holds the operational state for the rest of the app: appearance, browser language, cloud snapshot targets, camera selection with preview, microphone setup with live meters, delay and sync offsets, output quality profiles, recording defaults, minimal AI provider preferences, transport credentials, and onboarding restart. AI provider setup stays intentionally small: choose one active provider, enter only the endpoint or base URL when needed, the API key when needed, and the model or deployment names you actually use. Theme changes and layout preferences persist, and appearance changes propagate across tabs instead of drifting out of sync.
 
 ---
 
@@ -235,7 +235,7 @@ PrompterOne is in **active alpha**: the core authoring, rehearsal, reader, and l
 | **LiveKit transport** | 🟡 | Real transport-aware browser integration and guest-path work, with operational polish still expanding |
 | **Distribution routing** | 🟡 | Targets are capability-gated and blocked when the selected transport cannot service them |
 | **Cloud storage snapshots** | 🟡 | Browser-local provider configuration ships now; broader import/export maturity is still expanding |
-| **AI provider execution** | ✅ | AI Spotlight runs a configured Microsoft Agent Framework agent with route/editor context and MCP-style tools; script graph analysis uses the configured LLM extractor before any explicit tokenizer fallback |
+| **AI provider execution** | ✅ | AI Spotlight runs a configured Microsoft Agent Framework agent with route/editor context, MCP-style tools, structured chat-plus-edit output, and exact range edit application; script graph analysis uses the configured LLM extractor before any explicit tokenizer fallback |
 | **Generic RTMP fan-out** | ❌ | Intentionally unsupported without a real upstream transport path |
 | **PrompterOne backend** | ❌ | By design: the browser is the only app runtime |
 

src/PrompterOne.Core/AI/Agents/AssistantScriptAgent.cs

@@ -19,8 +19,11 @@ public sealed class AssistantScriptAgent : ScriptAgent
         You are the PrompterOne AI assistant.
         Answer the user's request using the active route, editor context, selected range, graph summary, and available tools.
         Use MCP-style tools when you need current document text, selected text, graph details, or app action metadata.
+        When a user asks to change specific wording, use the exact text search or range-reading tools to identify the source offsets before emitting documentEdits.
         For partial document edits, work through explicit range-based tool contracts. Do not regenerate a whole script unless the user clearly asks for a full rewrite.
-        If a change would mutate user text, propose a scoped edit and explain the reason clearly.
+        Return structured output with `chatMessage` for the visible chat response and `documentEdits` for machine-applied script edits.
+        Each document edit must use exact UTF-16 offsets with kind `insert`, `replace`, or `delete`; leave `documentEdits` empty when no document change is needed.
+        For every `replace` or `delete` edit, include `expectedText` exactly as it appears in the current source range. If you cannot identify an exact source range and expected text, do not emit a document edit; explain the blocker in `chatMessage`.
         Keep responses concise and directly useful inside the app.
         """;
 }

src/PrompterOne.Core/AI/Models/ScriptAgentRunResult.cs

@@ -5,4 +5,5 @@ public sealed record ScriptAgentRunResult(
     string WorkflowName,
     string Input,
     IReadOnlyList<ScriptAgentStepResult> Steps,
-    string Output);
+    string Output,
+    ScriptAgentStructuredOutput? StructuredOutput = null);

src/PrompterOne.Core/AI/Models/ScriptAgentStructuredOutput.cs

@@ -0,0 +1,42 @@
+using System.ComponentModel;
+
+namespace PrompterOne.Core.AI.Models;
+
+public sealed class ScriptAgentStructuredOutput
+{
+    [Description("Short text shown to the user in the AI Assistant chat surface.")]
+    public string ChatMessage { get; set; } = string.Empty;
+
+    [Description("Exact source document edits to apply after the chat response. Leave empty when no document change is needed.")]
+    public List<ScriptAgentStructuredEdit> DocumentEdits { get; set; } = [];
+}
+
+public sealed class ScriptAgentStructuredEdit
+{
+    [Description("Exclusive UTF-16 source offset where the edit ends.")]
+    public int End { get; set; }
+
+    [Description("The exact current source text in the range. Required for replace and delete edits.")]
+    public string ExpectedText { get; set; } = string.Empty;
+
+    [Description("Edit kind: insert, replace, or delete.")]
+    public string Kind { get; set; } = "replace";
+
+    [Description("Inclusive UTF-16 source offset where the edit starts.")]
+    public int Start { get; set; }
+
+    [Description("Inserted or replacement text. Leave empty for delete edits.")]
+    public string Text { get; set; } = string.Empty;
+
+    public ScriptDocumentEditOperation? ToDocumentEditOperation()
+    {
+        var normalizedKind = Kind.Trim().ToLowerInvariant();
+        return normalizedKind switch
+        {
+            "insert" => ScriptDocumentEditOperation.Insert(Start, Text),
+            "replace" => ScriptDocumentEditOperation.Replace(new ScriptDocumentRange(Start, End), Text),
+            "delete" => ScriptDocumentEditOperation.Delete(new ScriptDocumentRange(Start, End)),
+            _ => null
+        };
+    }
+}

src/PrompterOne.Core/AI/Services/ScriptAgentRuntime.cs

@@ -1,4 +1,6 @@
+using System.Text.Json;
 using Microsoft.Agents.AI;
+using Microsoft.Extensions.AI;
 using PrompterOne.Core.AI.Abstractions;
 using PrompterOne.Core.AI.Agents;
 using PrompterOne.Core.AI.Models;
@@ -11,6 +13,11 @@ public sealed class ScriptAgentRuntime(
     IEnumerable<ScriptWorkflow> workflows,
     IScriptAgentFactory agentFactory) : IScriptAgentRuntime
 {
+    private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web)
+    {
+        PropertyNameCaseInsensitive = true
+    };
+
     private readonly IReadOnlyDictionary<string, ScriptAgent> _agentsById = agents.ToDictionary(
         static agent => agent.Id,
         StringComparer.OrdinalIgnoreCase);
@@ -71,6 +78,31 @@ private static string ExtractOutput(AgentResponse response)
         return string.Join(Environment.NewLine + Environment.NewLine, messages);
     }
 
+    private static AgentRunOptions CreateStructuredOutputOptions() =>
+        new()
+        {
+            ResponseFormat = ChatResponseFormat.ForJsonSchema<ScriptAgentStructuredOutput>()
+        };
+
+    private static ScriptAgentStructuredOutput ExtractStructuredOutput(AgentResponse response)
+    {
+        var output = ExtractOutput(response);
+        var structuredOutput = JsonSerializer.Deserialize<ScriptAgentStructuredOutput>(output, JsonOptions);
+        if (structuredOutput is null)
+        {
+            throw new InvalidOperationException("The agent did not return the required structured output.");
+        }
+
+        structuredOutput.ChatMessage = structuredOutput.ChatMessage.Trim();
+        structuredOutput.DocumentEdits ??= [];
+        return structuredOutput;
+    }
+
+    private static string GetChatMessage(ScriptAgentStructuredOutput structuredOutput) =>
+        string.IsNullOrWhiteSpace(structuredOutput.ChatMessage)
+            ? "Done."
+            : structuredOutput.ChatMessage;
+
     private ScriptAgent GetRequiredAgent(string agentId) =>
         _agentsById.TryGetValue(agentId, out var agent)
             ? agent
@@ -98,6 +130,7 @@ private async Task<ScriptAgentRunResult> RunSequentialAsync(
     {
         var steps = new List<ScriptAgentStepResult>(workflow.AgentIds.Count);
         var nextInput = originalInput;
+        ScriptAgentStructuredOutput? finalStructuredOutput = null;
 
         foreach (var agentId in workflow.AgentIds)
         {
@@ -106,8 +139,10 @@ private async Task<ScriptAgentRunResult> RunSequentialAsync(
             var definition = GetRequiredAgent(agentId);
             var agent = await _agentFactory.CreateRequiredAsync(agentId, agentContext, cancellationToken);
             var session = await agent.CreateSessionAsync(cancellationToken);
-            var response = await agent.RunAsync(nextInput, session, new AgentRunOptions(), cancellationToken);
-            var output = ExtractOutput(response);
+            var response = await agent.RunAsync(nextInput, session, CreateStructuredOutputOptions(), cancellationToken);
+            var structuredOutput = ExtractStructuredOutput(response);
+            var output = GetChatMessage(structuredOutput);
+            finalStructuredOutput = structuredOutput;
 
             steps.Add(new ScriptAgentStepResult(definition.Id, definition.Name, nextInput, output));
             nextInput = BuildNextInput(originalInput, output);
@@ -118,7 +153,8 @@ private async Task<ScriptAgentRunResult> RunSequentialAsync(
             workflow.Name,
             originalInput,
             steps,
-            steps.Count == 0 ? string.Empty : steps[^1].Output);
+            steps.Count == 0 ? string.Empty : steps[^1].Output,
+            finalStructuredOutput);
     }
 
     private async Task<ScriptAgentRunResult> RunWorkflowAgentAsync(
@@ -129,14 +165,17 @@ private async Task<ScriptAgentRunResult> RunWorkflowAgentAsync(
     {
         var workflowAgent = await workflow.CreateWorkflowAgentAsync(_agentFactory, agentContext, cancellationToken);
         var session = await workflowAgent.CreateSessionAsync(cancellationToken);
-        var response = await workflowAgent.RunAsync(originalInput, session, new AgentRunOptions(), cancellationToken);
-        var output = ExtractOutput(response);
+        var response = await workflowAgent.RunAsync(originalInput, session, CreateStructuredOutputOptions(), cancellationToken);
+        var structuredOutput = ExtractStructuredOutput(response);
+        var output = GetChatMessage(structuredOutput);
 
         return new ScriptAgentRunResult(
             workflow.Id,
             workflow.Name,
             originalInput,
             [new ScriptAgentStepResult(workflow.Id, workflow.Name, originalInput, output)],
-            output);
+            output,
+            structuredOutput);
     }
+
 }

src/PrompterOne.Core/AI/Services/ScriptAgentToolProvider.cs

@@ -20,6 +20,7 @@ public IList<AITool> CreateTools(ScriptAgentContext? context)
             ScriptAgentMcpToolFunctionFactory.Create(contextTools.GetActivePrompterContext),
             ScriptAgentMcpToolFunctionFactory.Create(contextTools.ListAvailablePrompterOneTools),
             ScriptAgentMcpToolFunctionFactory.Create(contextTools.RequestPrompterOneTool),
+            ScriptAgentMcpToolFunctionFactory.Create(documentTools.FindScriptText),
             ScriptAgentMcpToolFunctionFactory.Create(documentTools.ReadScriptRange),
             ScriptAgentMcpToolFunctionFactory.Create(documentTools.ReadEditorSelection),
             ScriptAgentMcpToolFunctionFactory.Create(documentTools.ProposeScriptReplacement),