Skip to content

Commit cc14c98

Browse files
Merge pull request #51 from DataScience-EngineeringExperts/feat/elite-decision-protocol
feat: add Elite Decision Protocol
2 parents 4753b85 + b726458 commit cc14c98

31 files changed

Lines changed: 3580 additions & 291 deletions

‎CHANGELOG.md‎

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
### Added
11+
12+
- **Elite Decision Protocol (unreleased).** A quality-first `elite` mode for consequential
13+
decisions: independent member answers → concurrent council-wide claim audits → concurrent
14+
member revisions → the existing synthesis and canonical execution-traceable verdict. It is available
15+
through `--mode elite`, `Council.elite()`, and `elite_sync()` in source; no version, tag, or
16+
package publication is claimed here.
17+
- **Fixed three-success phase gate.** Each of Elite's `initial`, `critique`, and `revision`
18+
phases requires three successful responders. Larger councils may survive partial failures
19+
while three remain. A failed gate stops later calls and returns an incomplete result with a
20+
phase-specific reason, attempted artifacts, no synthesis/verdict, and CLI exit code 1.
21+
- **Execution-traceable results.** `EliteResult` preserves initial answers, critiques, and revisions;
22+
the manifest records a separate redacted receipt with `phase` provenance for every attempted
23+
buffered Elite call through synthesis, extraction, and repair, and aggregates reported
24+
usage/latency. Unknown cost remains `None` instead of being guessed.
25+
Elite is buffered-only: `--stream` is rejected before provider calls. The quality tradeoff is
26+
explicit—normally up to `3N + 2` calls for N members, or `3N + 3` when repair is attempted.
27+
- **Completion is not readiness.** `EliteResult.completed` records successful member phases;
28+
`decision_readiness` separately reports `ready`, `not_ready`, or `indeterminate` with reasons.
29+
The CLI exits 1 unless an Elite decision is `ready`.
30+
- **Version-aware cache identity.** Cache keys cover the resolved roster, generation and mode
31+
settings, extraction behavior, sanitized endpoint routing, optional source-bundle digest, and
32+
cache/protocol/prompt/schema versions. Old incompatible envelopes are safe misses.
33+
1034
### Fixed
1135

1236
- **`ModelHarnessManifest` now rides on *every* mode's result — a true invariant.**
@@ -26,6 +50,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2650

2751
### Documentation
2852

53+
- Added Elite CLI/library examples and reconciled the PDD, system diagram, README, and index
54+
around its unreleased status, fixed gate, partial-failure behavior, phased receipts, and
55+
latency/cost tradeoff.
2956
- Reconciled `README.md`, `SYSTEM_CONTEXT_DIAGRAM.md`, and the PDD so the
3057
manifest-on-every-result claim is now accurate, and documented the constrained-choice
3158
**`vote` mode** as **shipped** (CAC-09 / #3) rather than "absorbed by `provider_votes`"
@@ -34,9 +61,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3461

3562
## [1.1.0] - 2026-06-21
3663

37-
The **auditable council**. Every run now produces a structured, agreement-scored,
38-
fully auditable verdict plus a redacted execution manifest, on top of the existing
39-
synthesize/raw/debate/adversarial modes. The verdict is the product wedge: a
64+
The **auditable council** (the historical v1.1 release name). Every run produces a structured,
65+
agreement-scored, execution-traceable verdict plus a redacted execution manifest, on top of the
66+
existing synthesize/raw/debate/adversarial modes. The verdict is the product wedge: a
4067
multi-model council answer you can act on, with the agreement number computed by
4168
reproducible arithmetic over the model's clustering — never an LLM-emitted figure.
4269

‎DOCUMENTATION_INDEX.md‎

Lines changed: 23 additions & 14 deletions
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 85 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -12,19 +12,20 @@ It is **library-first** (the CLI is a thin shell over the same `Council` you imp
1212
returns **structured results** (per-model latency, token usage, and error capture), and is
1313
**partial-failure resilient** — one provider erroring never aborts the run. Keys are
1414
**bring-your-own**, referenced by environment-variable *name* only — never stored or
15-
logged. It ships four modes: **synthesize** (merge answers into one), **raw** (no merge),
15+
logged. Published v1.1 ships five modes: **synthesize** (merge answers into one), **raw** (no merge),
1616
**debate** (multi-round, members revise after seeing peers' anonymized answers), and
17-
**adversarial** (propose → refute → verdict). conclave is intentionally lightweight — a
18-
small council primitive, not an agent framework.
17+
**adversarial** (propose → refute → verdict), and **vote** (fixed-choice tally). This source
18+
branch also implements the unreleased **elite** quality-first claim-audit and revision mode.
19+
conclave is intentionally lightweight — a small council primitive, not an agent framework.
1920

20-
**The v1.1 wedge — the auditable council.** Every run also yields **a multi-model council
21-
verdict you can act on — structured, scored for agreement, fully auditable**: a
21+
**The v1.1 wedge — the execution-traceable council.** Decision/review synthesis runs can yield
22+
**a multi-model council verdict you can inspect — structured, scored for agreement, and
23+
execution-traceable**: a
2224
`CouncilVerdict` exposing agreement, `conflicts`, `minority_reports`, and `provider_votes`;
2325
a deterministic `consensus_score` (arithmetic over the model's clustering, *never* an
2426
LLM-emitted number); and a redacted `ModelHarnessManifest` recording how the run executed and
2527
which model produced the disagreement analysis. The verdict is **default-on**, and the manifest
26-
rides on **every** result — for all five modes (`synthesize`, `raw`, `debate`, `adversarial`,
27-
`vote`), not just `ask`. A constrained-choice **`vote` mode** (`--mode vote --choices ...`) also
28+
rides on **every** released-mode result; source-only Elite results carry it too. A constrained-choice **`vote` mode** (`--mode vote --choices ...`) also
2829
shipped in v1.1 (CAC-09 / #3) — distinct from the verdict's `provider_votes`, which score
2930
free-form agreement rather than tally a fixed ballot.
3031

@@ -125,17 +126,25 @@ conclave ask "Defend event sourcing for this ledger." \
125126
conclave ask "Which datastore for this workload?" \
126127
-c grok,gemini,claude --mode vote --choices "Postgres,DynamoDB,MongoDB"
127128

128-
# Machine-readable output (works for every mode; carries rounds/adversarial/vote too)
129+
# Elite (unreleased): independent answers -> claim audits -> revisions -> verdict
130+
conclave ask "Should we adopt a service mesh for 8 services?" \
131+
-c grok,gemini,claude --mode elite
132+
133+
# Machine-readable output (works for every mode; carries elite phase artifacts too)
129134
conclave ask "..." -c grok,perplexity --mode debate --json
130135
```
131136

132-
Mode flags at a glance: `--mode synthesize|raw|debate|adversarial|vote`. `--rounds N`
137+
Published v1.1 mode flags are `--mode synthesize|raw|debate|adversarial|vote`; this source
138+
branch additionally exposes `--mode elite`. `--rounds N`
133139
(default 2) is the *maximum* round count for `debate`; `--converge-threshold FLOAT`
134140
(or `--converge`/`--no-converge`) optionally stops a debate early once answers
135141
stabilize round-over-round (off by default — `--rounds` runs in full). `--proposer
136142
NAME` (default: first member) applies to `adversarial`. `--choices "A,B,C"` (two or
137143
more) is required for `vote`. `--synthesizer/-s` overrides the synthesizer *and* the
138-
adversarial judge. Every mode's result carries the auditable `ModelHarnessManifest`.
144+
adversarial judge and, on this source branch, Elite's final synthesizer. Every released mode's
145+
result carries the auditable `ModelHarnessManifest`; source-only Elite results do too. Elite is
146+
currently **unreleased**; use it from this source branch until
147+
a future release explicitly includes it.
139148

140149
`--council` accepts either a comma-separated list of friendly names or the name
141150
of a council defined in your config (see below). The built-in `default` council
@@ -152,6 +161,7 @@ Streaming and the non-streaming default produce the **same** final
152161
`CouncilResult`; `--stream` only changes how output is rendered. It is ignored
153162
with `--json` (which always emits the full structured payload), and on a cache
154163
hit the cached text is rendered in one shot rather than as a fake token stream.
164+
`--stream` is rejected for `elite` before any provider call; Elite runs are buffered only.
155165

156166
## Quickstart (library)
157167

@@ -172,10 +182,62 @@ for answer in result.answers:
172182
print("SYNTHESIS:\n", result.synthesis)
173183
```
174184

175-
## Auditable verdict
185+
### Elite Decision Protocol (unreleased)
186+
187+
Elite is the quality-first path for consequential decisions. It trades latency and provider
188+
spend for a stronger answer: three concurrent member phases, followed by conclave's existing
189+
synthesis and canonical structured verdict.
190+
191+
1. **Initial:** members answer independently.
192+
2. **Claim audit:** every surviving member audits the anonymized panel for supported,
193+
conflicting, and externally unverified claims. This compares answers; it does not check sources.
194+
3. **Revision:** survivors revise their own answer using the full anonymized panel and audits.
195+
4. **Final:** conclave synthesizes successful revisions, then applies the existing structured
196+
verdict and deterministic consensus calculation.
197+
198+
Each member phase has a fixed gate of **three successful responders**. A council larger than
199+
three may lose members and continue while at least three remain. If any gate falls below three,
200+
the protocol stops immediately: `result.elite.completed` is `False`, `failure_reason` names the
201+
failed phase, later phases are not called, and synthesis/verdict extraction do not run. Attempted
202+
answers and failures remain in `initial_answers`, `critiques`, or `revisions`; the CLI exits 1
203+
(and `--json` still emits the full result). This is stricter than the ordinary modes' best-effort
204+
partial-failure behavior by design.
205+
206+
```python
207+
from conclave import Council
208+
209+
council = Council(
210+
models=["grok", "gemini", "claude", "perplexity"],
211+
synthesizer="claude",
212+
)
213+
214+
result = council.elite_sync("Should we adopt a service mesh for 8 services?")
215+
# or: result = await council.elite("Should we adopt a service mesh for 8 services?")
216+
217+
if result.elite.completed and result.elite.decision_readiness == "ready":
218+
print(result.synthesis)
219+
print(result.verdict)
220+
else:
221+
print(result.elite.decision_readiness, result.elite.readiness_reasons)
222+
223+
for receipt in result.manifest.receipts:
224+
print(receipt.phase, receipt.name, receipt.outcome)
225+
```
226+
227+
Elite normally makes up to `3N + 2` calls for `N` members: three concurrent member phases,
228+
one synthesis call, and one verdict-extraction call; a schema repair can raise that to `3N + 3`.
229+
`Council(extract_verdict=False)` removes extraction and leaves readiness `indeterminate`. The
230+
secret-scanned manifest records every attempted member, synthesis, extraction, and repair call
231+
and aggregates reported latency/usage; unknown cost remains `None`. Elite does **not** stream.
232+
233+
`completed` means the three member phases ran successfully; it does not mean the output is ready
234+
to use. `decision_readiness` is separately `ready`, `not_ready`, or `indeterminate`, with
235+
machine-readable `readiness_reasons`; the CLI exits 1 unless readiness is `ready`.
236+
237+
## Execution-traceable verdict
176238

177239
On a decision- or review-style prompt with at least two responding members, conclave
178-
adjudicates the answers into a structured, agreement-scored, auditable **verdict**. It is
240+
adjudicates the answers into a structured, agreement-scored, execution-traceable **verdict**. It is
179241
**default-on** — no flag needed.
180242

181243
From the CLI, a decision prompt renders a green `VERDICT` panel:
@@ -232,16 +294,17 @@ print(result.manifest.verdict_extraction.model_id) # which model produced the
232294
print(result.manifest.secret_safety) # "verified_no_secrets"
233295
```
234296

235-
To opt out (e.g. for cost-sensitive runs — the verdict adds one extra synthesizer call per
236-
run), construct with `Council(extract_verdict=False)`; then `result.verdict` stays `None`.
297+
To opt out (e.g. for cost-sensitive runs — the verdict adds one initial synthesizer call and,
298+
when schema repair is needed, one retry), construct with `Council(extract_verdict=False)`;
299+
then `result.verdict` stays `None`.
237300
`conclave ask ... --json` already carries the full structured `verdict` and `manifest`.
238301

239302
**How the consensus number stays honest.** The single LLM-assisted step is *clustering* the
240303
members' stances; `consensus_score` is then deterministic arithmetic over that clustering
241304
(largest cluster / members with a position — no text-similarity, never model-emitted). Every
242305
cluster cites the member `answer_id`s backing it (`evidence_answer_ids`), and the manifest
243-
records which model + prompt version did the clustering, so the score is reproducible and
244-
traceable rather than a number to take on faith.
306+
records which model + prompt version did the clustering, so the calculation is traceable. It
307+
measures agreement over model-assisted clusters, not truth or external factual support.
245308

246309
**The verdict is optional.** It is absent — `result.verdict is None`, with the synthesis and
247310
member answers still returned — for an open-ended/creative prompt, for fewer than two
@@ -268,7 +331,7 @@ Event types: `member_delta` / `member_done` (per member, interleaved),
268331
A member that cannot stream, or any mid-stream failure, degrades gracefully —
269332
partial text is preserved and the error lands on that member's `ModelAnswer`,
270333
never raising (the same never-raises contract as `ask`). Streaming applies to
271-
`synthesize`/`raw` only; `debate`/`adversarial` are not streamed.
334+
`synthesize`/`raw` only; `debate`/`adversarial`/`vote`/`elite` are not streamed.
272335

273336
### Debate and adversarial modes
274337

@@ -364,11 +427,11 @@ Then: `conclave ask "..." --council fast`.
364427

365428
Repeated or eval runs can be served from an on-disk cache instead of re-calling
366429
the providers. It is **off by default** and **never persists API keys** — the
367-
cache key is a SHA-256 over the normalized prompt, the ordered council members
368-
(friendly name + resolved model id), the mode, the synthesizer/judge identity,
369-
and the mode params (temperature, debate `rounds` + `converge_threshold`,
370-
adversarial `proposer`). No key value or env-var name ever reaches the key or the
371-
stored payload.
430+
cache key is a SHA-256 over a canonical identity covering prompt, mode, ordered resolved
431+
models, synthesizer/judge, generation/mode settings, verdict-extraction behavior, sanitized
432+
custom-endpoint fingerprints, source-bundle digest, and cache/protocol/prompt/schema versions.
433+
No key value or raw endpoint URL reaches the identity or stored payload; incompatible older
434+
cache envelopes are misses, not migrated results.
372435

373436
Enable it per run with `--cache` (or disable a config default with `--no-cache`):
374437

0 commit comments

Comments
 (0)