diff --git a/README.md b/README.md index b2866908..d1461b6a 100644 --- a/README.md +++ b/README.md @@ -37,12 +37,13 @@ The course follows a four-week learning path: the scheduler does not rebuild dense history on every step. - **Week 4: Build a Coding Agent.** Start with a bounded, validated agent loop, then connect it to a small workspace. The course is publishing one reviewed - checkpoint at a time; Days 1 through 8 now cover inspection, approved edits, + checkpoint at a time; Days 1 through 9 now cover inspection, approved edits, one validation command, simple effect receipts, and one visible checkpoint-and-resume boundary, receipt-backed context compaction, and one visible inspect-and-steer pause, and deterministic evaluation of observable outcomes, then tokenizer/KV-prefix reuse for two isolated steered branches - and one explicit evidence-backed selection. + and one explicit evidence-backed selection, followed by bounded, + range-retrievable evidence for oversized tool results. ## Why MLX and Qwen3? @@ -75,7 +76,7 @@ implementation, test, and publication readiness is tracked below. ## Roadmap -The table tracks implementation (`Code`), tests (`Test`), rendered chapters (`Doc`), and Chi's review of learner-facing material (`Audit`). Week 4 is publishing one reviewed day at a time; Days 1 through 8 are currently available to learners. The Audit column reflects Chi's personal editorial pass on the published course content and is independent of code/test/doc readiness. +The table tracks implementation (`Code`), tests (`Test`), rendered chapters (`Doc`), and Chi's review of learner-facing material (`Audit`). Week 4 is publishing one reviewed day at a time; Days 1 through 9 are currently available to learners. The Audit column reflects Chi's personal editorial pass on the published course content and is independent of code/test/doc readiness. Day 3 can send file contents to the model, modify files after approval, and run one exact configured command. Use a disposable workspace without secrets and @@ -93,6 +94,9 @@ Day 8 reuses one real tokenizer/KV checkpoint for two differently steered, effect-isolated continuations, evaluates both with Day 7's harness, and makes one explicit passing selection without pretending completed effects were rewound. +Day 9 stores exact oversized tool-result bytes outside the model prompt, shows +a bounded identity/digest/head-tail observation, and lets the model retrieve +one explicit byte range through the existing loop. | Week + Chapter | Topic | Code | Test | Doc | Audit | |---|---|---|---|---|---| @@ -125,6 +129,7 @@ rewound. | 4.6 | Inspect and Steer a Paused Agent | βœ… | βœ… | βœ… | 🚧 | | 4.7 | Evaluate Observable Outcomes | βœ… | βœ… | βœ… | 🚧 | | 4.8 | Fork, Steer, and Select | βœ… | βœ… | βœ… | 🚧 | +| 4.9 | Bound Tool Evidence | βœ… | βœ… | βœ… | 🚧 | Other topics not covered include quantized or compressed KV caches, cross-request prefix caching, fine-tuning, and long-context techniques. diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index 7fd07ac7..06e46054 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -39,6 +39,7 @@ - [🚧 Day 6: Inspect and Steer a Paused Agent](./week4-06-steering.md) - [🚧 Day 7: Evaluate Observable Outcomes](./week4-07-evaluation.md) - [🚧 Day 8: Fork, Steer, and Select](./week4-08-fork-steer-select.md) + - [🚧 Day 9: Bound Tool Evidence](./week4-09-bound-tool-evidence.md) - [🚧 Appendix: Performance Evidence Ledger](./appendix-performance.md) - [Sponsored by Raft.build](./sponsor.md) diff --git a/book/src/week4-08-fork-steer-select.md b/book/src/week4-08-fork-steer-select.md index 803c5fcd..6cac9121 100644 --- a/book/src/week4-08-fork-steer-select.md +++ b/book/src/week4-08-fork-steer-select.md @@ -199,4 +199,8 @@ concurrently, share a mutable workspace, merge receipts, or provide a session server/tree. Day 8 teaches the boundary visibly before adding any serving-scale machinery. +Continue with [Day 9: Bound Tool Evidence](week4-09-bound-tool-evidence.md) to +keep oversized results verifiable without placing their complete bytes in each +later model prompt. + {{#include copyright.md}} diff --git a/book/src/week4-09-bound-tool-evidence.md b/book/src/week4-09-bound-tool-evidence.md new file mode 100644 index 00000000..71e784fe --- /dev/null +++ b/book/src/week4-09-bound-tool-evidence.md @@ -0,0 +1,236 @@ +# Day 9: Bound Tool Evidence + +An agent can read a log that is much larger than the useful part. Appending the +whole result to every later model prompt wastes context, but silently slicing +it loses the evidence needed to verify what happened. + +Day 9 keeps those two concerns separate: + +- preserve the exact UTF-8 tool-result bytes outside the prompt; +- give the model a bounded observation with identity, size, digest, and + head/tail previews; +- let the model request one explicit byte range and continue in the unchanged + agent loop. + +This is byte selection, not semantic summarization. The model decides which +range to inspect from visible facts. + +## Files You Implement + +| File | Public names | Responsibility | +| --- | --- | --- | +| `src/tiny_llm/agent/evidence.py` | `ArtifactRef`, `ArtifactStore`, `BoundedEvidenceWorkspace` | Store exact results, render bounded observations, and serve explicit ranges. | +| `src/tiny_llm/agent/__init__.py` | the names above | Export the cumulative Day 9 API. | + +The protocol, loop, workspace, receipts, and Days 1–8 modules do not change. +`BoundedEvidenceWorkspace` is a small adapter around the existing `Workspace`. + +Copy the Day 9 test into the learner workspace: + +```bash +pdm run copy-test --week 4 --day 9 +pdm run test --week 4 --day 9 +``` + +Before you implement the TODOs, the implementation-dependent cases across six +tasks are expected to fail; the shared constructor-validation cases already pass. + +## Task 1: Give Exact Bytes an Identity + +`ArtifactStore.put(result)` encodes the complete result as UTF-8, writes those +bytes under its explicit artifact root, and returns: + +```python +ArtifactRef( + artifact_id="artifact-", + byte_count=..., + sha256="", +) +``` + +The content-addressed ID and full digest deliberately repeat the same hash in +different roles: one is the handle used by the range request; the other is a +separately labeled model-visible verification field. The store registers the +ID in memory. A different store cannot retrieve it merely because the caller +guessed the filename. + +Before every range read, verify the stored byte count and digest again. The +course store is local and single-process. It does not promise retention, +garbage collection, encryption, access control, or a network blob service. It +preserves the exact bytes returned by the wrapped tool; earlier tool-level +limits, such as Day 3's command-output cap, still apply before this adapter. + +## Task 2: Replace Only Oversized Successful Results + +Wrap an existing workspace: + +```python +from tiny_llm.agent import ArtifactStore, BoundedEvidenceWorkspace + +bounded = BoundedEvidenceWorkspace( + workspace, + ArtifactStore(artifact_root), + max_inline_bytes=512, + preview_bytes=64, + max_range_bytes=512, +) +``` + +Short results and every `error:` observation remain byte-for-byte unchanged. +For a successful result larger than `max_inline_bytes`, persist the full bytes +and return a compact JSON observation containing: + +- `artifact_id`, `byte_count`, and `sha256`; +- valid UTF-8 head and tail previews with their byte ranges; +- the omitted half-open byte interval; +- one exact `read_file` range-request example. + +The entire compact observation, including metadata and previews, must fit +`max_inline_bytes`. Reduce previews at UTF-8 boundaries when the metadata needs +more space. Require `max_range_bytes >= 4` so the default range can always hold +one maximum-width UTF-8 code point. Never split a code point or silently replace +one. + +## Task 3: Reuse the Existing Tool Protocol + +Day 9 does not add a new action schema. It reserves one virtual relative-path +namespace for the existing `read_file` action: + +```text +.tool-artifacts//bytes/- +``` + +`[start,end)` is an exact half-open byte range. The adapter intercepts the +reserved prefix before the real workspace sees it. A successful reply names +the same artifact, total size, digest, start, end, returned byte count, and the +strictly decoded UTF-8 data. + +The reply is not sent back through externalization. Its selected data is +already limited by `max_range_bytes`. + +## Task 4: Fail Closed Without Leaking + +Every path beginning with `.tool-artifacts/` belongs to the virtual namespace. +Malformed paths must not fall through to a learner file of the same name. + +Return short ordinary `error:` observations for: + +- an invalid or unknown artifact ID; +- negative, reversed, out-of-bounds, or oversized ranges; +- stored bytes whose size or digest changed; +- a range that cuts through a UTF-8 code point. + +Do not print the host artifact-root path, enumerate known IDs, or reveal bytes +from another store while reporting an error. + +## Task 5: Continue Through the Same Loop + +The deterministic test creates a large ASCII build log whose diagnostic is +outside both previews. A scripted model performs three normal steps: + +```text +read_file build.log + | + v +bounded identity + previews + | + v +read_file .tool-artifacts//bytes/- + | + v +exact diagnostic range -> final answer +``` + +`run_agent` is unchanged. Its first event contains only the bounded +observation; the second contains only the selected range; the artifact file +still matches the complete original result. + +## Task 6: Preserve the Workspace Contract + +Delegate `policy`, `available_tools`, and `modified_files` to the wrapped +workspace. This lets `build_system_prompt`, action validation, and the existing +event loop operate without knowing about the storage adapter. + +The virtual range path is still a normal JSON `read_file` request, so the +learner does not need a second parser or a replacement generation interface. + +## Manual Cached-Qwen Walkthrough + +Complete the Day 9 TODOs first. Create separate disposable workspace and +artifact directories, put a large UTF-8 `build.log` in the workspace, and use +the same local-model adapter as the exploratory Week 4 exercise: + +```python +import hashlib +from pathlib import Path +from tempfile import TemporaryDirectory + +from mlx_lm import generate as mlx_generate, load +from tiny_llm.agent import ( + ArtifactStore, + BoundedEvidenceWorkspace, + ToolPolicy, + Workspace, + run_agent, +) + +workspace_directory = TemporaryDirectory(prefix="tiny-llm-day9-workspace-") +artifact_directory = TemporaryDirectory(prefix="tiny-llm-day9-artifacts-") +workspace_root = Path(workspace_directory.name) +artifact_root = Path(artifact_directory.name) +(workspace_root / "build.log").write_text( + "build started Ξ±\n" + + "x" * 256 + + "\nERROR code=E42 dependency mismatch\n" + + "y" * 3_000, + encoding="utf-8", +) + +mlx_model, tokenizer = load("Qwen/Qwen3-0.6B-MLX-4bit") +artifacts = ArtifactStore(artifact_root) +workspace = BoundedEvidenceWorkspace( + Workspace(ToolPolicy(workspace_root, max_file_bytes=64_000)), + artifacts, +) + +def generate(messages): + prompt = tokenizer.apply_chat_template( + messages, + tokenize=False, + add_generation_prompt=True, + enable_thinking=False, + ) + return mlx_generate( + mlx_model, tokenizer, prompt, max_tokens=256, verbose=False + ) + +run = run_agent( + "Read build.log. If it is externalized, retrieve one useful byte range.", + generate, + workspace, +) + +for event in run.events: + print(event.result) +print(run.final) +for artifact_path in artifact_root.iterdir(): + data = artifact_path.read_bytes() + print(artifact_path.name, len(data), hashlib.sha256(data).hexdigest()) +``` + +Model choices vary. Inspect the actual first observation, requested artifact +ID and range, returned bytes, final answer, and on-disk artifact digest. Do not +use a workspace or artifact root containing secrets. After inspection, call +`workspace_directory.cleanup()` and `artifact_directory.cleanup()`. + +## Checkpoint + +You can now keep a complete large tool result available for verification while +placing only bounded facts in the model context. The model can retrieve an +explicit range by identity and continue through the same tokenizer and agent +loop. + +Day 9 does not summarize the result, stream concurrent chunks, retain artifacts +for production, or add a network service. + +{{#include copyright.md}} diff --git a/book/src/week4-overview.md b/book/src/week4-overview.md index 4c744ad2..1ea044fa 100644 --- a/book/src/week4-overview.md +++ b/book/src/week4-overview.md @@ -1,7 +1,7 @@ # 🚧 Week 4: Build a Coding Agent > **Course status:** Week 4 is being published one checkpoint at a time. Days 1 -> through 8 are ready to learn and review. Additional capabilities will appear +> through 9 are ready to learn and review. Additional capabilities will appear > only after their implementation, starter, and reviews are ready. Weeks 1 through 3 turn tokens into text and make serving that text efficient. @@ -21,6 +21,9 @@ transcript shape. Day 8 reconnects the agent to the inference system from Weeks 1–3: it saves one real tokenizer/KV prefix, forks two isolated steered continuations without rewinding completed effects, evaluates both, and makes one explicit selection. +Day 9 keeps oversized tool-result bytes in a local artifact store while the +model sees a bounded identity, digest, preview, and explicit range-retrieval +path through the unchanged agent loop. ## What Day 1 Builds @@ -125,8 +128,15 @@ Select](week4-08-fork-steer-select.md). Its cumulative command is: pdm run test --week 4 --day 8 ``` -Only the Day 1 through Day 8 starter modules are published. Do not add session -trees, effect rewind, reconciliation, an LLM judge, radix serving, or other -later public APIs to your solution. +After Day 8 passes, continue with [Day 9: Bound Tool +Evidence](week4-09-bound-tool-evidence.md). Its cumulative command is: + +```bash +pdm run test --week 4 --day 9 +``` + +Only the Day 1 through Day 9 starter modules are published. Do not add session +trees, effect rewind, reconciliation, an LLM judge, semantic summarization, +radix serving, or other later public APIs to your solution. {{#include copyright.md}} diff --git a/docs/week4-day-split.md b/docs/week4-day-split.md index 37add743..ef7295ce 100644 --- a/docs/week4-day-split.md +++ b/docs/week4-day-split.md @@ -1,9 +1,9 @@ # Week 4 Day Split (reference for reviewers) -Status: Days 1--8 are published checkpoints. Each day ships as one cumulative +Status: Days 1--9 are published checkpoints. Each day ships as one cumulative learner PR so reviewers can see exactly what belongs to that checkpoint. -## 8-day structure +## 9-day structure | Day | Theme | Features (PRs) | Modules | |---|---|---|---| @@ -15,6 +15,7 @@ learner PR so reviewers can see exactly what belongs to that checkpoint. | 6 | Inspect and steer | safe-boundary status and one visible steering message | `steering.py` | | 7 | Evaluate outcomes | declared final/file/result/receipt facts | `evaluation.py` | | 8 | Fork, steer, and select | dense tokenizer/KV prefix reuse, isolated branches, explicit selection | `branching.py`, `workspace.py` | +| 9 | Bound tool evidence | exact external bytes, bounded observation, explicit range retrieval | `evidence.py` | Extension (not a day): COW/radix cache β€” `docs/week4-cow-radix-extension-plan.md`. @@ -26,10 +27,11 @@ Extension (not a day): COW/radix cache β€” `docs/week4-cow-radix-extension-plan. - Days are implemented sequentially. A later day does not leak API or prose into the current starter. -## Why 8 days +## Why 9 days The first seven days establish the agent loop and its observable evidence. Day 8 reconnects that control path to the tokenizer and KV cache built in Weeks -1--3. Each day adds one visible concept; scaling and production-hardening -machinery stay outside the core course unless a later checkpoint explicitly -teaches it. +1--3. Day 9 keeps large observable evidence available without filling every +later model prompt. Each day adds one visible concept; scaling and +production-hardening machinery stay outside the core course unless a later +checkpoint explicitly teaches it. diff --git a/src/tiny_llm/agent/__init__.py b/src/tiny_llm/agent/__init__.py index 9d9a6f28..dd378675 100644 --- a/src/tiny_llm/agent/__init__.py +++ b/src/tiny_llm/agent/__init__.py @@ -18,6 +18,7 @@ ResultExpectation, evaluate_run, ) +from .evidence import ArtifactRef, ArtifactStore, BoundedEvidenceWorkspace from .generation import generate_response, initial_messages from .loop import ( AgentEvent, @@ -47,6 +48,9 @@ "AgentRun", "AgentStatus", "ApprovalDecision", + "ArtifactRef", + "ArtifactStore", + "BoundedEvidenceWorkspace", "BranchOutcome", "CompactionResult", "EvaluationCase", diff --git a/src/tiny_llm/agent/evidence.py b/src/tiny_llm/agent/evidence.py new file mode 100644 index 00000000..779fff15 --- /dev/null +++ b/src/tiny_llm/agent/evidence.py @@ -0,0 +1,87 @@ +# WARNING: Under review - generated by LLM. + +"""Week 4, Day 9: keep large tool evidence outside the model context.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path + +from .protocol import ToolAction +from .workspace import ToolPolicy, Workspace + + +@dataclass(frozen=True) +class ArtifactRef: + """Identity and size of one exact externalized tool result.""" + + artifact_id: str + byte_count: int + sha256: str + + def __post_init__(self) -> None: + pass + + +@dataclass +class ArtifactStore: + """Store exact bytes under one explicit local artifact root.""" + + root: Path + _records: dict[str, ArtifactRef] = field(default_factory=dict, init=False) + + def __post_init__(self) -> None: + pass + + def put(self, result: str) -> ArtifactRef: + """Persist one complete UTF-8 tool result and return its identity.""" + + pass + + def read_range(self, artifact_id: str, start: int, end: int) -> bytes: + """Read one exact half-open byte range from a known artifact.""" + + pass + + def range_path(self, artifact_id: str, start: int, end: int) -> str: + """Return the virtual read_file path for one explicit byte range.""" + + pass + + +@dataclass +class BoundedEvidenceWorkspace: + """Bound model-visible results while retaining their exact external bytes.""" + + workspace: Workspace + artifacts: ArtifactStore + max_inline_bytes: int = 512 + preview_bytes: int = 64 + max_range_bytes: int = 512 + + def __post_init__(self) -> None: + if type(self.max_range_bytes) is not int or self.max_range_bytes < 4: + raise ValueError("max_range_bytes must be at least 4") + + @property + def policy(self) -> ToolPolicy: + """Expose the wrapped policy to the existing system prompt.""" + + pass + + @property + def available_tools(self) -> frozenset[str]: + """Reuse exactly the wrapped workspace's existing tool schema.""" + + pass + + @property + def modified_files(self) -> tuple[str, ...]: + """Expose file-tool changes from the wrapped workspace.""" + + pass + + def execute(self, action: ToolAction, tool_call_id: str | None = None) -> str: + """Execute normally, externalizing only oversized model observations.""" + + pass diff --git a/src/tiny_llm_ref/agent/__init__.py b/src/tiny_llm_ref/agent/__init__.py index 9d9a6f28..dd378675 100644 --- a/src/tiny_llm_ref/agent/__init__.py +++ b/src/tiny_llm_ref/agent/__init__.py @@ -18,6 +18,7 @@ ResultExpectation, evaluate_run, ) +from .evidence import ArtifactRef, ArtifactStore, BoundedEvidenceWorkspace from .generation import generate_response, initial_messages from .loop import ( AgentEvent, @@ -47,6 +48,9 @@ "AgentRun", "AgentStatus", "ApprovalDecision", + "ArtifactRef", + "ArtifactStore", + "BoundedEvidenceWorkspace", "BranchOutcome", "CompactionResult", "EvaluationCase", diff --git a/src/tiny_llm_ref/agent/evidence.py b/src/tiny_llm_ref/agent/evidence.py new file mode 100644 index 00000000..76c115b6 --- /dev/null +++ b/src/tiny_llm_ref/agent/evidence.py @@ -0,0 +1,259 @@ +# WARNING: Under review - generated by LLM. + +"""Week 4, Day 9: keep large tool evidence outside the model context.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import tempfile +from dataclasses import dataclass, field +from pathlib import Path + +from .protocol import AgentError, ToolAction +from .workspace import ToolPolicy, Workspace + +_ARTIFACT_ID = re.compile(r"artifact-[0-9a-f]{64}") +_RANGE_PATH = re.compile( + r"\.tool-artifacts/(artifact-[0-9a-f]{64})/bytes/([0-9]+)-([0-9]+)" +) + + +@dataclass(frozen=True) +class ArtifactRef: + """Identity and size of one exact externalized tool result.""" + + artifact_id: str + byte_count: int + sha256: str + + def __post_init__(self) -> None: + if not _ARTIFACT_ID.fullmatch(self.artifact_id): + raise ValueError("artifact id is invalid") + if type(self.byte_count) is not int or self.byte_count < 0: + raise ValueError("artifact byte count must be a non-negative integer") + if not re.fullmatch(r"[0-9a-f]{64}", self.sha256): + raise ValueError("artifact digest is invalid") + if self.artifact_id != f"artifact-{self.sha256}": + raise ValueError("artifact id must match its digest") + + +@dataclass +class ArtifactStore: + """Store exact bytes under one explicit local artifact root.""" + + root: Path + _records: dict[str, ArtifactRef] = field(default_factory=dict, init=False) + + def __post_init__(self) -> None: + root = Path(self.root) + if root.is_symlink(): + raise ValueError("artifact root must not be a symlink") + try: + root = root.resolve(strict=True) + except OSError as error: + raise ValueError("artifact root must exist") from error + if not root.is_dir(): + raise ValueError("artifact root must be a directory") + self.root = root + + def put(self, result: str) -> ArtifactRef: + """Persist one complete UTF-8 tool result and return its identity.""" + + if not isinstance(result, str): + raise ValueError("artifact result must be a string") + data = result.encode("utf-8") + digest = hashlib.sha256(data).hexdigest() + artifact_id = f"artifact-{digest}" + record = ArtifactRef(artifact_id, len(data), digest) + path = self.root / artifact_id + temporary: Path | None = None + try: + if not path.exists(): + with tempfile.NamedTemporaryFile(dir=self.root, delete=False) as output: + output.write(data) + temporary = Path(output.name) + os.replace(temporary, path) + elif path.read_bytes() != data: + raise AgentError("artifact contents conflict with their identity") + except AgentError: + raise + except OSError as error: + raise AgentError("could not store tool-result artifact") from error + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) + self._records[artifact_id] = record + return record + + def read_range(self, artifact_id: str, start: int, end: int) -> bytes: + """Read one exact half-open byte range from a known artifact.""" + + if not isinstance(artifact_id, str) or not _ARTIFACT_ID.fullmatch(artifact_id): + raise AgentError("artifact id is invalid") + if type(start) is not int or type(end) is not int: + raise AgentError("artifact range bounds must be integers") + if start < 0 or end <= start: + raise AgentError("artifact range must satisfy 0 <= start < end") + record = self._records.get(artifact_id) + if record is None: + raise AgentError("artifact is not available in this store") + try: + data = (self.root / artifact_id).read_bytes() + except OSError as error: + raise AgentError("could not read tool-result artifact") from error + if ( + len(data) != record.byte_count + or hashlib.sha256(data).hexdigest() != record.sha256 + ): + raise AgentError("artifact digest does not match its recorded identity") + if end > len(data): + raise AgentError("artifact range exceeds the stored byte count") + return data[start:end] + + def range_path(self, artifact_id: str, start: int, end: int) -> str: + """Return the virtual read_file path for one explicit byte range.""" + + if not isinstance(artifact_id, str) or not _ARTIFACT_ID.fullmatch(artifact_id): + raise ValueError("artifact id is invalid") + if type(start) is not int or type(end) is not int or start < 0 or end <= start: + raise ValueError("artifact range must satisfy 0 <= start < end") + return f".tool-artifacts/{artifact_id}/bytes/{start}-{end}" + + +@dataclass +class BoundedEvidenceWorkspace: + """Bound model-visible results while retaining their exact external bytes.""" + + workspace: Workspace + artifacts: ArtifactStore + max_inline_bytes: int = 512 + preview_bytes: int = 64 + max_range_bytes: int = 512 + + def __post_init__(self) -> None: + limits = (self.max_inline_bytes, self.preview_bytes, self.max_range_bytes) + if any(type(limit) is not int or limit <= 0 for limit in limits): + raise ValueError("evidence limits must be positive integers") + if self.max_inline_bytes < 512: + raise ValueError("max_inline_bytes must be at least 512") + if self.max_range_bytes < 4: + raise ValueError("max_range_bytes must be at least 4") + if self.preview_bytes > self.max_inline_bytes: + raise ValueError("preview_bytes must not exceed max_inline_bytes") + + @property + def policy(self) -> ToolPolicy: + """Expose the wrapped policy to the existing system prompt.""" + + return self.workspace.policy + + @property + def available_tools(self) -> frozenset[str]: + """Reuse exactly the wrapped workspace's existing tool schema.""" + + return self.workspace.available_tools + + @property + def modified_files(self) -> tuple[str, ...]: + """Expose file-tool changes from the wrapped workspace.""" + + return self.workspace.modified_files + + def execute(self, action: ToolAction, tool_call_id: str | None = None) -> str: + """Execute normally, externalizing only oversized model observations.""" + + if action.tool == "read_file": + raw_path = action.arguments.get("path") + if isinstance(raw_path, str) and raw_path.startswith(".tool-artifacts/"): + return self._read_artifact_path(raw_path) + result = self.workspace.execute(action, tool_call_id) + data = result.encode("utf-8") + if result.startswith("error:") or len(data) <= self.max_inline_bytes: + return result + record = self.artifacts.put(result) + return self._bounded_observation(record, data) + + def _bounded_observation(self, record: ArtifactRef, data: bytes) -> str: + _, suggested_end = self._head_preview(data, self.max_range_bytes) + for preview_limit in range(self.preview_bytes, -1, -1): + head, head_end = self._head_preview(data, preview_limit) + tail, tail_start = self._tail_preview(data, preview_limit, head_end) + payload = { + "artifact_id": record.artifact_id, + "byte_count": record.byte_count, + "head_preview": head, + "head_range": [0, head_end], + "omitted_range": [head_end, tail_start], + "range_request": { + "path": self.artifacts.range_path( + record.artifact_id, 0, suggested_end + ), + "tool": "read_file", + }, + "sha256": record.sha256, + "tail_preview": tail, + "tail_range": [tail_start, record.byte_count], + } + observation = "Tool result externalized:\n" + json.dumps( + payload, ensure_ascii=False, sort_keys=True, separators=(",", ":") + ) + if len(observation.encode("utf-8")) <= self.max_inline_bytes: + return observation + raise AgentError( + "evidence observation limit is too small for artifact metadata" + ) + + @staticmethod + def _head_preview(data: bytes, limit: int) -> tuple[str, int]: + end = min(limit, len(data)) + while end: + try: + return data[:end].decode("utf-8"), end + except UnicodeDecodeError as error: + end = error.start + return "", 0 + + @staticmethod + def _tail_preview(data: bytes, limit: int, head_end: int) -> tuple[str, int]: + start = max(head_end, len(data) - limit) + while start < len(data): + try: + return data[start:].decode("utf-8"), start + except UnicodeDecodeError as error: + start += max(1, error.end) + return "", len(data) + + def _read_artifact_path(self, raw_path: str) -> str: + match = _RANGE_PATH.fullmatch(raw_path) + if match is None: + return "error: artifact range path is malformed" + artifact_id, raw_start, raw_end = match.groups() + try: + start, end = int(raw_start), int(raw_end) + except ValueError: + return "error: artifact range path is malformed" + if end - start > self.max_range_bytes: + return f"error: artifact range exceeds {self.max_range_bytes} bytes" + try: + data = self.artifacts.read_range(artifact_id, start, end) + text = data.decode("utf-8") + except UnicodeDecodeError: + return "error: artifact range does not align to UTF-8 text" + except AgentError as error: + return f"error: {error}" + record = self.artifacts._records[artifact_id] + payload = { + "artifact_id": artifact_id, + "byte_count": len(data), + "data": text, + "end": end, + "sha256": record.sha256, + "start": start, + "total_byte_count": record.byte_count, + } + return "Artifact range:\n" + json.dumps( + payload, ensure_ascii=False, sort_keys=True, separators=(",", ":") + ) diff --git a/tests_refsol/test_week_4_day_9.py b/tests_refsol/test_week_4_day_9.py new file mode 100644 index 00000000..8212bdec --- /dev/null +++ b/tests_refsol/test_week_4_day_9.py @@ -0,0 +1,454 @@ +# WARNING: Under review - generated by LLM. + +"""Week 4 Day 9 bounded tool-evidence course-code tests.""" + +import hashlib +import json + +import pytest + +from .tiny_llm_base import ( + AgentLimits, + ArtifactRef, + ArtifactStore, + BoundedEvidenceWorkspace, + ToolAction, + ToolPolicy, + Workspace, + run_agent, +) + + +def _payload(observation: str, prefix: str) -> dict: + assert observation.startswith(prefix) + return json.loads(observation.removeprefix(prefix)) + + +def _workspace(tmp_path, content: str, **limits): + workspace_root = tmp_path / "workspace" + artifact_root = tmp_path / "artifacts" + workspace_root.mkdir() + artifact_root.mkdir() + (workspace_root / "build.log").write_text(content, encoding="utf-8") + workspace = Workspace(ToolPolicy(workspace_root, max_file_bytes=32_000)) + artifacts = ArtifactStore(artifact_root) + return BoundedEvidenceWorkspace(workspace, artifacts, **limits), artifacts + + +def test_task_1_artifact_store_preserves_exact_bytes_and_identity(tmp_path): + artifact_root = tmp_path / "artifacts" + artifact_root.mkdir() + store = ArtifactStore(artifact_root) + result = "Ξ±\n" + "build output\n" * 40 + data = result.encode("utf-8") + + record = store.put(result) + + digest = hashlib.sha256(data).hexdigest() + assert record == ArtifactRef(f"artifact-{digest}", len(data), digest) + assert (artifact_root / record.artifact_id).read_bytes() == data + assert store.read_range(record.artifact_id, 2, 17) == data[2:17] + assert store.put(result) == record + with pytest.raises(ValueError, match="artifact id"): + ArtifactRef("artifact-UPPER", len(data), digest) + with pytest.raises(ValueError, match="byte count"): + ArtifactRef(record.artifact_id, -1, digest) + with pytest.raises(ValueError, match="digest"): + ArtifactRef(record.artifact_id, len(data), "not-a-digest") + other_digest = hashlib.sha256(b"other").hexdigest() + with pytest.raises(ValueError, match="must match"): + ArtifactRef(record.artifact_id, len(data), other_digest) + + +def test_task_2_oversized_result_becomes_bounded_verifiable_observation(tmp_path): + content = "HEAD----" + "x" * 3_000 + "----TAIL" + bounded, artifacts = _workspace( + tmp_path, + content, + max_inline_bytes=512, + preview_bytes=8, + max_range_bytes=24, + ) + + observation = bounded.execute(ToolAction("read_file", {"path": "build.log"})) + payload = _payload(observation, "Tool result externalized:\n") + + data = content.encode() + digest = hashlib.sha256(data).hexdigest() + artifact_id = f"artifact-{digest}" + assert payload == { + "artifact_id": artifact_id, + "byte_count": len(data), + "head_preview": "HEAD----", + "head_range": [0, 8], + "omitted_range": [8, len(data) - 8], + "range_request": { + "path": artifacts.range_path(artifact_id, 0, 24), + "tool": "read_file", + }, + "sha256": digest, + "tail_preview": "----TAIL", + "tail_range": [len(data) - 8, len(data)], + } + assert len(observation.encode()) <= bounded.max_inline_bytes + assert len(observation) < len(content) + assert (artifacts.root / artifact_id).read_bytes() == data + + unicode_content = "Ξ±Ξ²Ξ³" + "x" * 1_000 + "δΡ΢" + unicode_root = tmp_path / "unicode" + unicode_root.mkdir() + unicode_artifacts = tmp_path / "unicode-artifacts" + unicode_artifacts.mkdir() + (unicode_root / "build.log").write_text(unicode_content, encoding="utf-8") + unicode = BoundedEvidenceWorkspace( + Workspace(ToolPolicy(unicode_root, max_file_bytes=32_000)), + ArtifactStore(unicode_artifacts), + max_inline_bytes=512, + preview_bytes=5, + max_range_bytes=24, + ) + rendered = unicode.execute(ToolAction("read_file", {"path": "build.log"})) + unicode_payload = _payload(rendered, "Tool result externalized:\n") + assert "οΏ½" not in unicode_payload["head_preview"] + assert "οΏ½" not in unicode_payload["tail_preview"] + assert len(rendered.encode()) <= unicode.max_inline_bytes + + +def test_task_2_externalization_uses_utf8_bytes_at_the_exact_inline_boundary( + tmp_path, +): + inline, _ = _workspace(tmp_path, "a" * 512) + assert inline.execute(ToolAction("read_file", {"path": "build.log"})) == "a" * 512 + + multibyte_root = tmp_path / "multibyte" + multibyte_root.mkdir() + multibyte_artifacts = tmp_path / "multibyte-artifacts" + multibyte_artifacts.mkdir() + content = "Γ©" * 512 + (multibyte_root / "build.log").write_text(content, encoding="utf-8") + multibyte = BoundedEvidenceWorkspace( + Workspace(ToolPolicy(multibyte_root, max_file_bytes=32_000)), + ArtifactStore(multibyte_artifacts), + ) + + result = multibyte.execute(ToolAction("read_file", {"path": "build.log"})) + payload = _payload(result, "Tool result externalized:\n") + assert payload["byte_count"] == len(content.encode("utf-8")) == 1_024 + + +def test_task_2_inline_cap_plus_one_byte_is_externalized(tmp_path): + bounded, _ = _workspace(tmp_path, "a" * 513) + + result = bounded.execute(ToolAction("read_file", {"path": "build.log"})) + + assert result.startswith("Tool result externalized:\n") + + +def test_task_2_four_byte_unicode_bounds_the_whole_encoded_observation(tmp_path): + bounded, _ = _workspace( + tmp_path, + "πŸ™‚" * 300, + max_inline_bytes=512, + preview_bytes=80, + max_range_bytes=512, + ) + + result = bounded.execute(ToolAction("read_file", {"path": "build.log"})) + + assert len(result.encode("utf-8")) <= 512 + + +def test_task_2_four_byte_unicode_preview_ranges_are_byte_accurate(tmp_path): + content = "πŸ™‚" * 300 + bounded, _ = _workspace( + tmp_path, + content, + max_inline_bytes=512, + preview_bytes=80, + max_range_bytes=512, + ) + + result = bounded.execute(ToolAction("read_file", {"path": "build.log"})) + payload = _payload(result, "Tool result externalized:\n") + + data = content.encode("utf-8") + head_end = payload["head_range"][1] + tail_start = payload["tail_range"][0] + assert head_end == len(payload["head_preview"].encode("utf-8")) + assert tail_start == len(data) - len(payload["tail_preview"].encode("utf-8")) + assert payload["omitted_range"] == [head_end, tail_start] + + +def test_task_3_explicit_virtual_read_returns_only_the_bound_range(tmp_path): + content = "0123456789" * 100 + bounded, artifacts = _workspace( + tmp_path, + content, + max_inline_bytes=512, + preview_bytes=5, + max_range_bytes=32, + ) + first = bounded.execute(ToolAction("read_file", {"path": "build.log"})) + identity = _payload(first, "Tool result externalized:\n") + path = artifacts.range_path(identity["artifact_id"], 117, 139) + + result = bounded.execute(ToolAction("read_file", {"path": path})) + payload = _payload(result, "Artifact range:\n") + + assert payload == { + "artifact_id": identity["artifact_id"], + "byte_count": 22, + "data": content.encode()[117:139].decode(), + "end": 139, + "sha256": identity["sha256"], + "start": 117, + "total_byte_count": len(content.encode()), + } + assert content[:117] not in result + assert content[139:] not in result + + +def test_task_3_selected_unicode_byte_count_is_not_a_character_count(tmp_path): + content = "€" * 300 + bounded, artifacts = _workspace(tmp_path, content) + first = bounded.execute(ToolAction("read_file", {"path": "build.log"})) + identity = _payload(first, "Tool result externalized:\n") + path = artifacts.range_path(identity["artifact_id"], 0, 510) + + result = bounded.execute(ToolAction("read_file", {"path": path})) + payload = _payload(result, "Artifact range:\n") + + assert payload["byte_count"] == 510 + assert len(payload["data"]) == 170 + + +def test_task_3_advertised_unicode_range_is_aligned_and_runnable(tmp_path): + content = "€" * 300 + bounded, _ = _workspace(tmp_path, content) + first = bounded.execute(ToolAction("read_file", {"path": "build.log"})) + identity = _payload(first, "Tool result externalized:\n") + + advertised = identity["range_request"]["path"] + assert advertised.endswith("/bytes/0-510") + result = bounded.execute(ToolAction("read_file", {"path": advertised})) + payload = _payload(result, "Artifact range:\n") + assert payload["start"] == 0 and payload["end"] == 510 + assert payload["data"] == "€" * 170 + + +@pytest.mark.parametrize("max_range_bytes", [1, 2, 3]) +def test_task_3_range_cap_must_hold_one_max_width_utf8_character( + tmp_path, max_range_bytes +): + with pytest.raises(ValueError, match="max_range_bytes must be at least 4"): + _workspace(tmp_path, "πŸ™‚" * 130, max_range_bytes=max_range_bytes) + + +def test_task_3_four_byte_range_cap_is_advertised_and_runnable(tmp_path): + bounded, _ = _workspace(tmp_path, "πŸ™‚" * 130, max_range_bytes=4) + first = bounded.execute(ToolAction("read_file", {"path": "build.log"})) + identity = _payload(first, "Tool result externalized:\n") + + advertised = identity["range_request"]["path"] + assert advertised.endswith("/bytes/0-4") + result = bounded.execute(ToolAction("read_file", {"path": advertised})) + payload = _payload(result, "Artifact range:\n") + assert payload["start"] == 0 and payload["end"] == 4 + assert payload["byte_count"] == 4 and payload["data"] == "πŸ™‚" + + +def test_task_3_four_byte_cap_still_rejects_a_supplied_misaligned_range(tmp_path): + bounded, artifacts = _workspace(tmp_path, "πŸ™‚" * 130, max_range_bytes=4) + record = artifacts.put("πŸ™‚" * 130) + misaligned = artifacts.range_path(record.artifact_id, 0, 3) + + result = bounded.execute(ToolAction("read_file", {"path": misaligned})) + + assert result == "error: artifact range does not align to UTF-8 text" + + +@pytest.mark.parametrize( + ("path", "message"), + [ + (".tool-artifacts/not-an-id/bytes/0-4", "path is malformed"), + ("{artifact}/bytes/-1-2", "path is malformed"), + ("{artifact}/bytes/9-2", "0 <= start < end"), + ("{artifact}/bytes/0-99", "exceeds 16 bytes"), + ("{artifact}/bytes/0-16", "stored byte count"), + ], +) +def test_task_4_range_failures_are_ordinary_and_do_not_fall_through( + tmp_path, path, message +): + bounded, artifacts = _workspace( + tmp_path, + "short", + max_inline_bytes=512, + preview_bytes=4, + max_range_bytes=16, + ) + record = artifacts.put("short") + path = path.format(artifact=f".tool-artifacts/{record.artifact_id}") + learner_path = bounded.workspace.policy.root / path + learner_path.parent.mkdir(parents=True, exist_ok=True) + learner_path.write_text("learner-secret", encoding="utf-8") + + result = bounded.execute(ToolAction("read_file", {"path": path})) + + assert result.startswith("error: ") + assert message in result + assert "short" not in result + assert "learner-secret" not in result + + +@pytest.mark.parametrize("oversized_bound", ["{digits}-1", "0-{digits}"]) +def test_task_4_oversized_decimal_bounds_are_ordinary_and_do_not_fall_through( + tmp_path, monkeypatch, oversized_bound +): + bounded, artifacts = _workspace(tmp_path, "short") + record = artifacts.put("short") + digits = "9" * 5_000 + path = f".tool-artifacts/{record.artifact_id}/bytes/" + oversized_bound.format( + digits=digits + ) + + def fail_if_delegated(*_args, **_kwargs): + raise AssertionError("reserved artifact paths must not reach the workspace") + + monkeypatch.setattr(bounded.workspace, "execute", fail_if_delegated) + + result = bounded.execute(ToolAction("read_file", {"path": path})) + + assert result == "error: artifact range path is malformed" + + +def test_task_4_unknown_store_and_tampering_do_not_leak_artifacts(tmp_path): + bounded, artifacts = _workspace( + tmp_path, + "classified diagnostic", + max_inline_bytes=512, + preview_bytes=4, + max_range_bytes=16, + ) + record = artifacts.put("classified diagnostic") + path = artifacts.range_path(record.artifact_id, 0, 10) + + unrelated_root = tmp_path / "unrelated" + unrelated_root.mkdir() + unrelated = BoundedEvidenceWorkspace( + bounded.workspace, + ArtifactStore(unrelated_root), + max_inline_bytes=512, + preview_bytes=4, + max_range_bytes=16, + ) + missing = unrelated.execute(ToolAction("read_file", {"path": path})) + assert missing == "error: artifact is not available in this store" + assert record.artifact_id not in missing + assert "classified" not in missing + + unicode_record = artifacts.put("Ξ±Ξ²Ξ³") + split_path = artifacts.range_path(unicode_record.artifact_id, 1, 3) + split = bounded.execute(ToolAction("read_file", {"path": split_path})) + assert split == "error: artifact range does not align to UTF-8 text" + + (artifacts.root / record.artifact_id).write_text("tampered", encoding="utf-8") + tampered = bounded.execute(ToolAction("read_file", {"path": path})) + assert tampered == "error: artifact digest does not match its recorded identity" + assert "classified" not in tampered + + +def test_task_4_range_cap_plus_one_is_rejected(tmp_path): + bounded, artifacts = _workspace( + tmp_path, + "a" * 100, + max_inline_bytes=512, + preview_bytes=4, + max_range_bytes=16, + ) + record = artifacts.put("a" * 100) + path = artifacts.range_path(record.artifact_id, 0, 17) + + result = bounded.execute(ToolAction("read_file", {"path": path})) + + assert result == "error: artifact range exceeds 16 bytes" + + +def test_task_5_existing_agent_loop_retrieves_evidence_then_continues(tmp_path): + diagnostic = "ERROR code=E42 dependency mismatch" + content = "head\n" + "x" * 500 + diagnostic + "\n" + "y" * 500 + "\ntail" + bounded, artifacts = _workspace( + tmp_path, + content, + max_inline_bytes=512, + preview_bytes=8, + max_range_bytes=64, + ) + digest = hashlib.sha256(content.encode()).hexdigest() + artifact_id = f"artifact-{digest}" + start = content.encode().index(diagnostic.encode()) + end = start + len(diagnostic.encode()) + responses = iter( + ( + '{"tool":"read_file","path":"build.log"}', + json.dumps( + { + "tool": "read_file", + "path": artifacts.range_path(artifact_id, start, end), + }, + separators=(",", ":"), + ), + '{"final":"The requested range contains diagnostic E42."}', + ) + ) + + run = run_agent( + "Inspect the large build log and retrieve the diagnostic range.", + lambda _messages: next(responses), + bounded, + AgentLimits(max_steps=3), + ) + + assert run.completed and run.final == "The requested range contains diagnostic E42." + assert len(run.events) == 3 + assert diagnostic not in run.events[0].result + range_payload = _payload(run.events[1].result, "Artifact range:\n") + assert range_payload["artifact_id"] == artifact_id + assert range_payload["start"] == start and range_payload["end"] == end + assert range_payload["data"] == diagnostic + assert (artifacts.root / artifact_id).read_bytes() == content.encode() + + +def test_task_6_small_and_error_results_remain_inline_and_state_is_delegated( + tmp_path, monkeypatch +): + bounded, artifacts = _workspace( + tmp_path, + "small result\n", + max_inline_bytes=512, + preview_bytes=8, + max_range_bytes=16, + ) + + result = bounded.execute(ToolAction("read_file", {"path": "build.log"})) + + assert result == "small result\n" + assert bounded.policy is bounded.workspace.policy + assert bounded.available_tools == bounded.workspace.available_tools + assert bounded.modified_files == () + assert list(artifacts.root.iterdir()) == [] + + long_error = "error: " + "x" * 1_000 + monkeypatch.setattr(bounded.workspace, "execute", lambda *_args: long_error) + assert bounded.execute(ToolAction("read_file", {"path": "missing"})) == long_error + assert list(artifacts.root.iterdir()) == [] + + with pytest.raises(ValueError, match="at least 512"): + BoundedEvidenceWorkspace(bounded.workspace, artifacts, max_inline_bytes=511) + with pytest.raises(ValueError, match="must not exceed"): + BoundedEvidenceWorkspace( + bounded.workspace, + artifacts, + max_inline_bytes=512, + preview_bytes=513, + ) diff --git a/tests_refsol/test_week_4_starter_sync.py b/tests_refsol/test_week_4_starter_sync.py index 9e9b7744..c552395c 100644 --- a/tests_refsol/test_week_4_starter_sync.py +++ b/tests_refsol/test_week_4_starter_sync.py @@ -1,6 +1,6 @@ # WARNING: Under review - generated by LLM. -"""Course-code guards for the cumulative Week 4 Day 1--8 starter.""" +"""Course-code guards for the cumulative Week 4 Day 1--9 starter.""" import ast import importlib.util @@ -22,6 +22,7 @@ "branching", "checkpoint", "compaction", + "evidence", "evaluation", "generation", "loop", @@ -127,7 +128,7 @@ def test_starter_is_solution_free_and_reference_is_implemented(module): ) -def test_package_exports_match_the_published_day_8_surface(): +def test_package_exports_match_the_published_day_9_surface(): import tiny_llm.agent as starter import tiny_llm_ref.agent as refsol @@ -139,6 +140,9 @@ def test_package_exports_match_the_published_day_8_surface(): "AgentRun", "AgentStatus", "ApprovalDecision", + "ArtifactRef", + "ArtifactStore", + "BoundedEvidenceWorkspace", "BranchOutcome", "CompactionResult", "EvaluationCase", @@ -174,12 +178,13 @@ def test_package_exports_match_the_published_day_8_surface(): assert set(starter.__all__) == set(refsol.__all__) == expected -def test_only_day_1_through_day_8_modules_exist_in_the_starter(): +def test_only_day_1_through_day_9_modules_exist_in_the_starter(): allowed = { "__init__.py", "branching.py", "checkpoint.py", "compaction.py", + "evidence.py", "evaluation.py", "generation.py", "loop.py", @@ -305,10 +310,60 @@ def test_day_8_branching_surface_is_complete_and_has_no_future_api(): source = "\n".join( path.read_text(encoding="utf-8") for path in STARTER.glob("*.py") ) - for future_name in ("Session", "reconcile", "radix", "ArtifactStore"): + for future_name in ("Session", "reconcile", "radix"): assert future_name not in source +def test_day_9_bounded_evidence_surface_is_complete_and_has_no_future_api(): + evidence = _public_surface(STARTER, "evidence") + assert set(evidence) == { + "ArtifactRef", + "ArtifactStore", + "BoundedEvidenceWorkspace", + } + assert {name for name, *_ in evidence["ArtifactStore"]} == { + "put", + "range_path", + "read_range", + } + assert {name for name, *_ in evidence["BoundedEvidenceWorkspace"]} == { + "available_tools", + "execute", + "modified_files", + "policy", + } + source = "\n".join( + path.read_text(encoding="utf-8") for path in STARTER.glob("*.py") + ) + for future_name in ("SemanticSummary", "BlobService", "StreamingDispatcher"): + assert future_name not in source + + +def test_day_9_starter_range_cap_can_hold_one_max_width_utf8_character(tmp_path): + from tiny_llm.agent import ( + ArtifactStore, + BoundedEvidenceWorkspace, + ToolPolicy, + Workspace, + ) + + workspace_root = tmp_path / "workspace" + artifact_root = tmp_path / "artifacts" + workspace_root.mkdir() + artifact_root.mkdir() + workspace = Workspace(ToolPolicy(workspace_root)) + artifacts = ArtifactStore(artifact_root) + + for max_range_bytes in (1, 2, 3): + with pytest.raises(ValueError, match="max_range_bytes must be at least 4"): + BoundedEvidenceWorkspace( + workspace, artifacts, max_range_bytes=max_range_bytes + ) + + bounded = BoundedEvidenceWorkspace(workspace, artifacts, max_range_bytes=4) + assert bounded.max_range_bytes == 4 + + def test_removed_catalog_hash_is_not_exported_or_declared(): for root in (STARTER, REFSOL): source = (root / "protocol.py").read_text(encoding="utf-8")