feat(agentic-harness): local runnable agentic API-orchestration harness on ReVoman - #407
Open
overfullstack wants to merge 50 commits into
Open
feat(agentic-harness): local runnable agentic API-orchestration harness on ReVoman#407overfullstack wants to merge 50 commits into
overfullstack wants to merge 50 commits into
Conversation
New agentic-harness Gradle module (koog + :revoman) that makes every production-system-design.md concept runnable locally, no Salesforce org: deterministic ReVoman spine, tool-def gen, router/slot-filler, evals + confusion matrix, OTel spans, HITL confirm gate + flywheel. 4 runnable stages; STUB LLM in CI, real Claude gated on ANTHROPIC_API_KEY.
Implements Prompt A (Router) and Prompt B (SlotFiller) for the agentic harness orchestrator. SlotFiller validates every argument against the tool's typed SlotSchema BEFORE returning, rejecting hallucinated names, out-of-enum values, non-integer numbers, and missing required slots.
Add Stage 3 Task 1: EvalSet (snakeyaml loader) + ConfusionMatrix with accuracy/render. - router-eval.yaml: 7 labeled cases with deliberate near-miss (price phrased with "quote") - EvalCase: simple data class for utterance + expected graph label - EvalSet: loads YAML resource via snakeyaml, returns List<EvalCase> - ConfusionMatrix: counts correct/off-diagonal predictions; render() produces labeled grid - ConfusionMatrices.from(): builder for (expected, predicted) pairs; null -> "none"
Implement two evaluation idioms: BFCL-style AST comparison of predicted (graph, slots) against gold YAML, and tau-bench-style final database state validation after orchestration.
…e_tool) Implements GenAiTracer, a dependency-free span model for OpenTelemetry GenAI conventions with console rendering. Spans nest via block scope using ArrayDeque stack; child spans become children of their parent via try/finally semantics. Includes Span data class with render() for indented tree output, NoopTracer for silent operation, and convention helpers (invokeAgentAttrs/chatAttrs/ executeToolAttrs) for semantic span attributes.
…atch, loop close Implements the full feedback flywheel with observability: 1. GenAiTracer + instrumented Orchestrator emit OTel GenAI-convention spans (invoke_agent, chat, execute_tool) with gen_ai.* attributes to console. 2. ConfirmGate turns proposals into feedback labels (Positive, Negative, CorrectionPair) via scripted confirm/reject decisions. 3. NightlyBatch mines labels to grow the eval set and draft when_not_to_use clauses from confusion pairs (rejected near-miss turns). 4. The loop closes: re-evaluating the router on the grown set with the auto-drafted clauses shows accuracy improve (6/7 → 7/7 in the demo), proving the "CI gate tightens itself" flywheel works end-to-end. FlywheelClosesLoopTest asserts that a rejected near-miss (quote vs price) → drafted clause → strictly higher accuracy on the grown eval set. Also fixes ScoringLlmClient's penalty logic to require ≥2 matching trigger tokens (not just ANY) to prevent overly-broad penalties on legitimate uses. Runnable: ./gradlew :agentic-harness:runStage4Demo -q
…ness) Completes the 6 reliability scaffolds from reasoning-layer.md as a delta on the existing module: confidence on RouteDecision, retrieval pre-filter, the confidence/disambiguation gate (ask-don't-guess), a confirm-gate preview object, end-to-end ReasoningLayer wiring, and native Anthropic tool-use for the real client. Reuses the ~60% already built (tool-def gen, slot-filler, evals, confusion matrix).
…me + dataFlow) Implements GraphContractWriter with toJson(Verbosity) extension function that renders a unified contract (metadata descriptor + runtime outcome) at three tiers: - SUMMARY: version/graph/slots/stopReason (no descriptive prose) - STANDARD: adds whenToUse/whenNotToUse and dataFlow lineage - VERBOSE: adds exampleQueries and full nested Rundown JSON Hand-built JSON string with defensive quote/backslash escaping per brief.
…additive (decouple StubJudge)
…(declared vs actual)
…+ parse-validity test
Implements RL Task 3: the three core types for the reasoning layer's disambiguation gate. ConfidencePolicy handles tunable thresholds (write graphs default to 0.90 financial-services band, reads 0.60), ActionPreview carries proposed-action details for confirmation, and ReasoningOutcome is the sealed type for the gate's decision (NoMatch, Clarify, ConfirmRequired, Proceed).
…margin 0.5 < 0.90)
… etc.) for live demo The claude source set inherited main's compiled output but not its dependencies, so the live ClaudeDemo failed loading OAS/graphs (snakeyaml). Add main's compile/runtime classpaths. Quarantine intact: default build still never compiles this set (verified: :agentic-harness:test green, no compileClaudeKotlin). Live Bedrock-proxy call remains best-effort: the SFDC gateway is reachable (HTTP 307) but koog's Bedrock client does not complete a call against it (redirect/protocol) — the deterministic layer via StubLlmClient is the shippable deliverable.
…gLlmClient type is deliberate
Qodana for JVM8 new problems were found
💡 Qodana analysis was run in the pull request mode: only the changed files were checked Contact Qodana teamContact us at qodana-support@jetbrains.com
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| testing { | ||
| suites { | ||
| getByName<JvmTestSuite>("test") { useJUnitJupiter(libs.versions.junit.get()) } |
|
|
||
| testing { | ||
| suites { | ||
| getByName<JvmTestSuite>("test") { useJUnitJupiter(libs.versions.junit.get()) } |
| */ | ||
| package com.salesforce.revoman.harness | ||
|
|
||
| import com.salesforce.revoman.harness.GraphRunner |
|
|
||
| import com.salesforce.revoman.harness.mock.MockCpqServer | ||
| import com.salesforce.revoman.harness.orchestrator.GraphRegistry | ||
| import com.salesforce.revoman.harness.reasoning.ConfidencePolicy |
| * call (graph name + filled slots) against gold by value equality. Rejected (invalid) slot-fills | ||
| * predict no slots — a clean fail, never a silent partial pass. | ||
| */ | ||
| class BfclCheck(private val llm: LlmClient, private val tools: List<ToolDef>) { |
| class Orchestrator( | ||
| private val baseUrl: String, | ||
| private val tools: List<ToolDef>, | ||
| private val llm: LlmClient, |
| // ScoringLlmClient populates. A bare LlmClient leaves margin=0.0, causing all decisions to fall | ||
| // below threshold and always route to Clarify. | ||
| private val llm: ScoringLlmClient = ScoringLlmClient(), | ||
| private val policy: ConfidencePolicy = ConfidencePolicy(), |
| val text = | ||
| javaClass.classLoader.getResourceAsStream(resource)?.bufferedReader()?.readText() | ||
| ?: error("Slot-fill gold not found on classpath: $resource") | ||
| @Suppress("UNCHECKED_CAST") val root = Yaml().load<Map<String, Any?>>(text) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds
agentic-harness— a new, isolated Gradle module that makes the agentic API-orchestration design (production-system-design / reasoning-layer) runnable locally, end-to-end, with no Salesforce org. It demonstrates every production concept as working, tested code on top of ReVoman as the deterministic execution engine.The ReVoman library is not modified. The only change outside
agentic-harness/is a singleinclude("agentic-harness")line insettings.gradle.kts. New deps (koog) are quarantined to an opt-in source set the default build never compiles.What's in it
Built in stages, each independently runnable (
./gradlew :agentic-harness:runStage*Demo):configure/price/quote) + three ReVoman V3 collections chaining via{{var}}, executed byReVoman.revUp, with a Layer-1 contract test on theRundown.LlmClient(deterministic stub + real Claude), router + slot-filler with schema validation before execution, and the orchestrator-workers loop.when_not_to_useclause → accuracy 6/7 → 7/7), BFCL slot-fill check, tau-bench final-DB-state check, bounded LLM-as-judge alongside a deterministic ground-truth.invoke_agent/chat/execute_tool), a HITL confirm gate (confirm/edit/reject → labels), and a nightly-batch that grows the eval set + draftswhen_not_to_usefrom rejects.GraphContractfuses the metadata descriptor (ToolDef) + runtime outcome (Rundown) + data-lineage (which{{var}}came from which step) + a version envelope, with a tiered JSON renderer; plusContractFidelityCheck(deterministic API-drift catch) andContractAblationEval(measure accuracy delta from contract enrichment).{graph, confidence, margin}, retrieval pre-filter, the confidence/disambiguation gate (ask-don't-guess: low margin or bad slots → clarify; writes → confirm-required, never auto-execute), and end-to-endReasoningLayerwiring.Isolation & safety
src/claude(a source set the defaulttest/build/checknever compiles). Grep-verified: noai.koog.*/kotlinx.coroutines.*undersrc/mainorsrc/test.Verification
Runnable demos (deterministic, no key):
Known limit
The real Claude path compiles and targets this environment's AWS Bedrock proxy; the live call is best-effort (the gateway is reachable but koog's Bedrock client does not complete a request against the proxy's redirect/protocol). The deterministic stub path is the shippable deliverable and exercises the entire design in CI.
Design + implementation notes live under
docs/superpowers/specs/anddocs/superpowers/plans/.🤖 Generated with Claude Code