Skip to content

Commit 4753b85

Browse files
fix(manifest): attach ModelHarnessManifest on every mode's result (#50)
The auditable ModelHarnessManifest was documented as first-class on every CouncilResult ("rides on every result, not behind a debug flag"), but only the ask/synthesize path built one. debate, adversarial, and vote constructed their result directly in modes.py and returned with manifest=None, so the invariant silently drifted for 3 of 5 modes and no test guarded it. Make manifest attachment a true invariant with a single-site fix: all five modes funnel through Council._cached_run, which now calls a new Council._ensure_manifest on every returned result. It is a no-op for the synthesize/raw path (which builds its own richer manifest in _ask_uncached so it can populate verdict-provenance slots and re-stamp secret-safety), and fills the manifest for debate/adversarial/vote from the resolved membership + collected answers. This also covers the zero-members early return and the cache-hit path, so no result can escape without a secret-safety-VERIFIED manifest. Verdict scope is deliberately unchanged: the clustering verdict still runs on synthesize/ask only. It is not forced onto adversarial (which already emits a judge verdict, so it would double-count the same adjudication) and is deferred on debate/vote; rationale recorded in PDD 4a. Add regression tests (tests/test_manifest_all_modes.py) asserting manifest is not None and secret_safety == verified_no_secrets for debate, adversarial, and vote, including the zero-members and cache-hit paths. Reconcile docs so (a) the manifest-on-every-result claim is now true and (b) the constrained-choice vote mode is documented as shipped (CAC-09 / #3), not "absorbed by provider_votes": README, SYSTEM_CONTEXT_DIAGRAM, PDD, DOCUMENTATION_INDEX, and CHANGELOG.
1 parent 91a81a9 commit 4753b85

7 files changed

Lines changed: 441 additions & 77 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 31 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,33 @@ All notable changes to conclave are documented here.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [Unreleased]
9+
10+
### Fixed
11+
12+
- **`ModelHarnessManifest` now rides on *every* mode's result — a true invariant.**
13+
The auditable manifest was documented as first-class on every `CouncilResult`, but
14+
`debate`, `adversarial`, and `vote` built their result directly in `modes.py` and
15+
returned with `manifest = None` (only `ask`/synthesize attached one). The fix moves
16+
manifest attachment to the single chokepoint every mode funnels through
17+
(`Council._cached_run` → new `Council._ensure_manifest`): it fills the manifest from
18+
the resolved membership + collected answers for `debate`/`adversarial`/`vote`
19+
(including the zero-members early return and cache hits), is a no-op for the
20+
synthesize/raw path that already builds its own richer manifest, and stamps
21+
`secret_safety = verified_no_secrets` when the manifest is provably clean. Regression
22+
tests (`tests/test_manifest_all_modes.py`) pin the invariant per mode so it cannot
23+
drift again. The clustering **verdict** scope is unchanged: it still runs on
24+
`synthesize`/`ask` only and is intentionally not layered onto `adversarial` (which
25+
already emits a judge verdict) — see PDD §4a.
26+
27+
### Documentation
28+
29+
- Reconciled `README.md`, `SYSTEM_CONTEXT_DIAGRAM.md`, and the PDD so the
30+
manifest-on-every-result claim is now accurate, and documented the constrained-choice
31+
**`vote` mode** as **shipped** (CAC-09 / #3) rather than "absorbed by `provider_votes`"
32+
— the two are complementary (a fixed ballot vs. clustered free-form stances), not the
33+
same feature. Added the `--mode vote --choices` CLI example and a §4a verdict-scope note.
34+
835
## [1.1.0] - 2026-06-21
936

1037
The **auditable council**. Every run now produces a structured, agreement-scored,
@@ -58,10 +85,10 @@ reproducible arithmetic over the model's clustering — never an LLM-emitted fig
5885

5986
### Note
6087

61-
- **`vote` mode (council issue #3) is absorbed/superseded** by the verdict work:
62-
`provider_votes` records which provider took which position (with evidence) and
63-
`consensus_label`/`consensus_score` report the split deterministically — no separate
64-
`vote` mode is shipped or planned.
88+
- **`vote` (council issue #3):** at 1.1.0 the verdict's `provider_votes` +
89+
`consensus_label`/`consensus_score` were considered to subsume it. **Superseded post-1.1.0**
90+
— a real constrained-choice `vote` mode later shipped (CAC-09 / #3; see `[Unreleased]`),
91+
complementary to `provider_votes`: a fixed ballot vs. clustered free-form stances.
6592

6693
## [1.0.0] - 2026-06-14
6794

‎DOCUMENTATION_INDEX.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ the canonical authority spec on top of those.
2323

2424
| Doc | Path | Purpose |
2525
|-----|------|---------|
26-
| **Product Design Document** | [`docs/PRODUCT_DESIGN_DOCUMENT.md`](docs/PRODUCT_DESIGN_DOCUMENT.md) | **Canonical** product spec and roadmap, reframed in v1.1 to **the auditable multi-model council**. Problem & vision, personas, BYO-keys/key-handling security, council modes (synthesize/raw/debate/adversarial; `vote` absorbed by the verdict), the v1.1 auditable verdict (§4a — CouncilResult v2, deterministic consensus, native structured output, the manifest), provider matrix, architecture, scope + non-goals, demand-gated v1.2 roadmap, the mcp-warden dev-time boundary, licensing & positioning, open questions. **When docs disagree, the PDD wins.** |
26+
| **Product Design Document** | [`docs/PRODUCT_DESIGN_DOCUMENT.md`](docs/PRODUCT_DESIGN_DOCUMENT.md) | **Canonical** product spec and roadmap, reframed in v1.1 to **the auditable multi-model council**. Problem & vision, personas, BYO-keys/key-handling security, council modes (synthesize/raw/debate/adversarial/vote — `vote` shipped as a constrained-choice ballot, CAC-09 / #3, complementary to the verdict's `provider_votes`), the v1.1 auditable verdict (§4a — CouncilResult v2, deterministic consensus, native structured output, the manifest), provider matrix, architecture, scope + non-goals, demand-gated v1.2 roadmap, the mcp-warden dev-time boundary, licensing & positioning, open questions. **When docs disagree, the PDD wins.** |
2727

2828
---
2929

@@ -81,7 +81,7 @@ Run: `pytest` (config in `pyproject.toml`, `asyncio_mode = "auto"`).
8181
|------|------|---------|
8282
| Packaging | [`pyproject.toml`](pyproject.toml) | hatchling build, deps (httpx, pydantic, rich, typer, pyyaml — no LLM SDK), dev extras, console script, pytest config. License: MIT. **PyPI distribution name `conclave-cli`** (the name `conclave` is an unrelated project); command + import stay `conclave`. |
8383
| Release runbook | [`RELEASING.md`](RELEASING.md) | Operator runbook: one-time PyPI OIDC Trusted-Publisher setup for `conclave-cli`, cut-a-release checklist (bump→tag→publish Release), post-release verification (Sigstore bundle, PEP 740 attestations), rollback/yank. |
84-
| Changelog | [`CHANGELOG.md`](CHANGELOG.md) | Keep-a-Changelog history per release (SemVer). The `[1.1.0]` (unreleased) entry covers the auditable council — CouncilResult v2, deterministic consensus, native structured output, the manifest, and the absorbed `vote` mode; the 1.0.0 entry covers the distribution rename, key-leak hardening, synthesizer versioning, and release engineering. |
84+
| Changelog | [`CHANGELOG.md`](CHANGELOG.md) | Keep-a-Changelog history per release (SemVer). The `[Unreleased]` entry covers the manifest-on-every-result invariant fix and the `vote`-mode-shipped doc reconciliation; `[1.1.0]` covers the auditable council — CouncilResult v2, deterministic consensus, native structured output, the manifest; the 1.0.0 entry covers the distribution rename, key-leak hardening, synthesizer versioning, and release engineering. |
8585
| Dev lockfile | [`requirements-dev.lock`](requirements-dev.lock) | Hash-pinned dev + runtime tree for reproducible installs/CI. Regenerate via `uv pip compile --universal --generate-hashes --python-version 3.11 --extra dev pyproject.toml -o requirements-dev.lock`. |
8686
| License | [`LICENSE`](LICENSE) | MIT License. Copyright (c) 2026 Ernest Provo. Matches the `pyproject.toml` license field. |
8787
| Security policy | [`SECURITY.md`](SECURITY.md) | BYO-keys vulnerability reporting policy: how to report, scope, and the key-handling guarantees consumers can rely on. |
@@ -100,6 +100,7 @@ Run: `pytest` (config in `pyproject.toml`, `asyncio_mode = "auto"`).
100100

101101
| Date | Change |
102102
|------|--------|
103+
| 2026-07-13 | **Manifest-on-every-result invariant fix + doc reconciliation.** `ModelHarnessManifest` now attaches on all five modes (was `None` for `debate`/`adversarial`/`vote`) via a single-site fix at `Council._cached_run` → new `_ensure_manifest`; regression tests in `tests/test_manifest_all_modes.py`. Docs corrected so the manifest-on-every-result claim is accurate and the `vote` mode is documented as **shipped** (CAC-09 / #3), not "absorbed" — across README, `SYSTEM_CONTEXT_DIAGRAM.md`, PDD (§1/§3/§4a/§8/§9/§12), and `CHANGELOG.md` (`[Unreleased]`). Verdict scope unchanged (still `synthesize`/`ask`-only; not layered on `adversarial`). |
103104
| 2026-06-21 | **v1.1 docs pivot — the auditable multi-model council.** PDD reframed to the auditable-council wedge (new §4a: CouncilResult v2, deterministic `position_cluster_ratio_v1` consensus, native + fallback structured output, the verdict-optional rule, the secret-free `ModelHarnessManifest`); `vote` mode marked **absorbed** by `provider_votes` across PDD §1/§4/§8/§9/§12; demand-gated v1.2 "Operable Council" roadmap. System Context diagram gains the verdict pipeline + manifest (mermaid re-validated). README gains an "Auditable verdict" section (CLI panel + library example). `CHANGELOG.md` `[1.1.0]` added. v0.x mode-detail prose archived to [`docs/archive/pdd-v0.x-modes-detail.md`](docs/archive/pdd-v0.x-modes-detail.md) to keep the PDD under 500 lines. New modules documented: `verdict.py`, `agreement.py`, `verdict_synthesis.py`, `manifest.py`. |
104105
| 2026-06-14 | v1.0.0 release. Version bump 0.3.0 → 1.0.0; new `CHANGELOG.md` (Keep-a-Changelog). Integrates the three v1.0 PRs below. |
105106
| 2026-06-14 | v1.0 distribution + release-engineering (PR-A): PyPI distribution name → `conclave-cli` (command + import stay `conclave`); OIDC Trusted-Publishing release workflow (`.github/workflows/release.yml`, SHA-pinned, PEP 740 attestations, Sigstore keyless signing, inert until a Release fires + publisher configured); `pip-audit` fail-closed CI job in `test.yml`; hash-pinned `requirements-dev.lock`; `RELEASING.md` operator runbook. |

‎README.md‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,11 @@ verdict you can act on — structured, scored for agreement, fully auditable**:
2222
`CouncilVerdict` exposing agreement, `conflicts`, `minority_reports`, and `provider_votes`;
2323
a deterministic `consensus_score` (arithmetic over the model's clustering, *never* an
2424
LLM-emitted number); and a redacted `ModelHarnessManifest` recording how the run executed and
25-
which model produced the disagreement analysis. The verdict is **default-on**. (A `vote` mode
26-
was once on the roadmap; it is **absorbed** by `provider_votes` — see below.)
25+
which model produced the disagreement analysis. The verdict is **default-on**, and the manifest
26+
rides on **every** result — for all five modes (`synthesize`, `raw`, `debate`, `adversarial`,
27+
`vote`), not just `ask`. A constrained-choice **`vote` mode** (`--mode vote --choices ...`) also
28+
shipped in v1.1 (CAC-09 / #3) — distinct from the verdict's `provider_votes`, which score
29+
free-form agreement rather than tally a fixed ballot.
2730

2831
See the canonical spec and design docs:
2932

@@ -118,16 +121,21 @@ conclave ask "Is a service mesh worth it for 8 services?" \
118121
conclave ask "Defend event sourcing for this ledger." \
119122
-c grok,gemini,perplexity --mode adversarial --proposer grok
120123

121-
# Machine-readable output (works for every mode; carries rounds/adversarial too)
124+
# Vote: each member picks one labelled choice; plurality winner (or split) is tallied
125+
conclave ask "Which datastore for this workload?" \
126+
-c grok,gemini,claude --mode vote --choices "Postgres,DynamoDB,MongoDB"
127+
128+
# Machine-readable output (works for every mode; carries rounds/adversarial/vote too)
122129
conclave ask "..." -c grok,perplexity --mode debate --json
123130
```
124131

125-
Mode flags at a glance: `--mode synthesize|raw|debate|adversarial`. `--rounds N`
132+
Mode flags at a glance: `--mode synthesize|raw|debate|adversarial|vote`. `--rounds N`
126133
(default 2) is the *maximum* round count for `debate`; `--converge-threshold FLOAT`
127134
(or `--converge`/`--no-converge`) optionally stops a debate early once answers
128135
stabilize round-over-round (off by default — `--rounds` runs in full). `--proposer
129-
NAME` (default: first member) applies to `adversarial`. `--synthesizer/-s` overrides
130-
the synthesizer *and* the adversarial judge.
136+
NAME` (default: first member) applies to `adversarial`. `--choices "A,B,C"` (two or
137+
more) is required for `vote`. `--synthesizer/-s` overrides the synthesizer *and* the
138+
adversarial judge. Every mode's result carries the auditable `ModelHarnessManifest`.
131139

132140
`--council` accepts either a comma-separated list of friendly names or the name
133141
of a council defined in your config (see below). The built-in `default` council

‎SYSTEM_CONTEXT_DIAGRAM.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ flowchart TB
2929
cli["CLI · conclave ask / providers (cli.py)"]
3030
lib["Library API · from conclave import Council (__init__.py)"]
3131
council["Council orchestrator<br/>fan_out · synthesize_blocks · skip-no-key (council.py)"]
32-
modes["Deliberation modes<br/>debate · adversarial (modes.py + prompts.py)"]
32+
modes["Deliberation modes<br/>debate · adversarial · vote (modes.py + prompts.py)"]
3333
registry["Registry · name to model-id<br/>key PRESENCE only, never values (registry.py)"]
3434
config["Config loader · custom endpoints (config.py)"]
3535
models["Result contract · CouncilResult v2<br/>answers · verdict · consensus · manifest (models.py)"]
@@ -147,7 +147,11 @@ flowchart TB
147147
with a prompt-level fallback for providers without strict support. The **`ModelHarnessManifest`**
148148
(`manifest.py`) rides on **every** result — first-class, not a debug flag — recording which
149149
model + prompt version produced the clustering (provenance) and stamping `secret_safety`
150-
only after the serialized manifest is scanned clean. A verdict is *optional*: open-ended
150+
only after the serialized manifest is scanned clean. This is enforced as a true invariant at
151+
the single chokepoint `Council._cached_run` → `_ensure_manifest`: **all five modes**
152+
(`synthesize`, `raw`, `debate`, `adversarial`, `vote`) funnel through it, so a result can
153+
never escape without a manifest — including the zero-members early return and cache hits.
154+
A verdict is *optional*: open-ended
151155
prompts, fewer than two responding members, or extraction failure leave `verdict = None`
152156
with the synthesis and member answers intact and the reason recorded on the manifest.
153157
- **Streaming shares the same boundary (PDD §9 #5).** A `--stream` run (and the library

0 commit comments

Comments
 (0)