feat(plugin): cover goroutine mode and exact replay for v0.2.0

plugins/claude-code/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "faultline",
-  "version": "0.1.2",
+  "version": "0.2.0",
   "description": "Debug, replay and pin failing seeds of faultline simulation tests, deterministic simulation testing for distributed Go code.",
   "author": {
     "name": "Hamed Yousefi"

plugins/claude-code/agents/triage.md

@@ -36,7 +36,12 @@ The task names an artifact directory, or a seed such as `0x287372ab06f1482e`.
 
 For a determinism failure (the headline starts with `determinism failure`), the report names the
 first differing record and both versions of it. Find that record in `timeline.txt` and follow its
-cause to the choice that came from outside the seed.
+cause to the choice that came from outside the seed. If the report mentions coverage gaps, read
+`toolexec-gaps.txt` and list the gaps in the code under test: they are the first places to look.
+
+If the directory holds only `stall.txt`, a goroutine-mode run stalled and the watchdog ended the
+test binary. Read its first lines (the virtual time, the last event, the replay command) and the
+goroutine states it marks without `(durable)`. Put `stall` as the kind on the `check:` line.
 
 ## Answer
 

plugins/claude-code/evals/flaky-test-without-faultline/graders/offers-faultline-test.md

@@ -2,5 +2,5 @@
 type: llm
 ---
 
-PASS if the reply says that faultline cannot run this test or the replication code as it is, because the code starts goroutines and uses `net` (it needs a port to faultline's node API or simulated network first), and offers to write a faultline simulation test of the replication.
-FAIL if the reply says or implies that faultline can replay the existing TestReplicasCatchUp from a seed unchanged (for example by setting FAULTLINE_SEED on it), if it does not offer a faultline test, or if it does not mention faultline.
+PASS if the reply says that faultline cannot replay the existing test or run the replication code as it is, because the code calls `net.Listen` or `net.Dial` itself (it must take its listener and dialer as values, such as faultline's simulated network `p.Net` in goroutine mode, or be ported to faultline's node API), and offers to write a faultline simulation test of the replication.
+FAIL if the reply says or implies that faultline can replay the existing TestReplicasCatchUp from a seed unchanged (for example by setting FAULTLINE_SEED on it), if it says that faultline cannot run goroutine code at all, if it does not offer a faultline test, or if it does not mention faultline.

plugins/claude-code/evals/flaky-test-without-faultline/prompt.md

@@ -1,5 +1,5 @@
 ---
-description: "The faultline skill loads for a flaky replication test built on goroutines and net with no faultline test, and the reply says faultline v0.1.x cannot run that code unchanged and offers a faultline test. Needs --scaffold."
+description: "The faultline skill loads for a flaky replication test built on goroutines and net with no faultline test, and the reply says the code must take its listener and dialer as values before faultline can run it in goroutine mode, and offers a faultline test. Needs --scaffold."
 tags: [trigger]
 max_turns: 10
 allowed_tools: [Read, Glob, Grep, Skill]

plugins/claude-code/skills/faultline/SKILL.md

@@ -11,6 +11,12 @@ injected faults. Every random choice comes from one seed, so a failing seed repl
 run: the same check fails at the same virtual time and event number. faultline settings are
 `FAULTLINE_*` environment variables, never test flags.
 
+A test with `Options.Mode: faultline.ModeGoroutine` runs ordinary goroutine code as processes
+(`w.AddProcess`). Its report ends with a `determinism:` line: `exact (toolexec backend)` when the
+test binary was built with `-toolexec=faultline-toolexec`, else `level 3 (goroutine mode,
+best-effort replay)`, where the Go runtime still picks goroutine order, map order and `select`
+cases, so a replay usually but not always fails the same way.
+
 When `superpowers:systematic-debugging` or `superpowers:test-driven-development` is loaded, this
 loop is their reproduce, root-cause and red-green steps for a faultline test. It needs neither.
 
@@ -35,12 +41,13 @@ below.
    code runs the same seeds as CI. A CI job with `FAULTLINE_EXPLORE=1` ran fresh seeds: take the
    seed and the replay command from its log.
 
-   A faultline seed does not fail by chance: on the same code and Go minor version, it fails
-   again on every run. So a re-run that passed ran other seeds (`-short` caps the count;
+   A faultline seed does not fail by chance: on the same code and Go minor version, an event-mode
+   or exact seed fails again on every run. So a re-run that passed ran other seeds (`-short` caps the count;
    `FAULTLINE_EXPLORE`, `FAULTLINE_BASE_SEED`, `FAULTLINE_SEED` and `FAULTLINE_SEEDS` change the
-   list), ran changed code or another Go minor version, or the test is not deterministic. Check
-   the last case with `FAULTLINE_CHECK_DETERMINISM=1 go test -run '^TestKV$' .`. Never tell the
-   user that a random fault happened to miss the re-run.
+   list), ran changed code or another Go minor version, ran a best-effort goroutine-mode seed, or
+   the test is not deterministic. Check the last two cases with
+   `FAULTLINE_CHECK_DETERMINISM=1 go test -run '^TestKV$' .`. Never tell the user that a random
+   fault happened to miss the re-run.
 
 2. **Triage in a subagent.** Give the `artifacts:` path, or the seed, to the `faultline:triage`
    agent and ask why the seed failed. It returns the failed check, the events that led to it, the
@@ -50,7 +57,10 @@ below.
 
 3. **Reproduce.** Run the replay command (the user's `/faultline:replay <seed>` does the same). It
    must fail the same check at the same virtual time and event before you change any code. Any
-   other result is a determinism problem: read [rules.md](rules.md).
+   other result is a determinism problem: read [rules.md](rules.md). Keep
+   `-toolexec=faultline-toolexec` when the command has it; if the build stops with
+   `run "faultline-toolexec prepare"`, run `faultline-toolexec prepare` in the module and replay
+   again.
 
 4. **Fix the code under test**, following [rules.md](rules.md). Never weaken, skip or delete an
    invariant or final check to make a seed pass.
@@ -101,6 +111,14 @@ Done when the replay passes, fails again with the fix undone, both `go test` run
 For etcd/raft, use `etcdraft.Test` or `etcdraft.Setup` from the
 [harness guide](https://raw.githubusercontent.com/hmdsefi/faultline/main/docs/guides/etcd-raft.md).
 
+For code that starts goroutines, serves `net/http` or gRPC, or writes files, write a goroutine-mode
+test instead, from
+[Test goroutine code](https://raw.githubusercontent.com/hmdsefi/faultline/main/docs/guides/goroutine-mode.md):
+set `Mode: faultline.ModeGoroutine`, add the service with `w.AddProcess`, and pass it `p.Net` and
+`p.FS` in place of its listener, dialer and filesystem. It needs faultline v0.2.0 or later in
+`go.mod`. For exact replay, build it through `faultline-toolexec`, from
+[Replay goroutine code exactly](https://raw.githubusercontent.com/hmdsefi/faultline/main/docs/guides/exact-replay.md).
+
 Done when `go test ./...` and `FAULTLINE_CHECK_DETERMINISM=1 go test ./...` print `ok`, and the
 planted bug made a seed fail.
 
@@ -119,14 +137,16 @@ Its failures do not replay from a seed. If the module already has a faultline te
 code, use the loop above with that test. Otherwise, answer the user's question about the flaky
 test, and in the same reply:
 
-- Say plainly what faultline v0.1.x cannot run unchanged: code that starts goroutines, uses `net`
-  or opens files with `os` needs a port to the node API first ([rules.md](rules.md) has the rules
-  for ported code). etcd/raft v3.7.0 runs through `harness/etcdraft` (`etcdraft.Test`), which does
-  that port.
+- Say plainly that faultline cannot replay the existing test: it uses the real network, disk and
+  clock. Say what the code needs to run inside faultline. Goroutine code runs in goroutine mode
+  (faultline v0.2.0 or later) once it takes its listener, dialer and filesystem as values instead
+  of calling `net.Listen`, `net.Dial` or `os.Open` itself; name the functions that call them.
+  etcd/raft v3.7.0 runs through `harness/etcdraft` (`etcdraft.Test`).
 - Offer to write a faultline test of the behavior the flaky test checks, following "Write a new
   simulation test" above. Write it only when the user agrees.
 
-Done when the reply states that limit for this code and offers the faultline test.
+Done when the reply states what this code needs before faultline can run it and offers the
+faultline test.
 
 ## Reference
 

plugins/claude-code/skills/faultline/artifacts.md

@@ -22,8 +22,10 @@ same directory.
   jq '{signature: .failure.signature, at: .failure.at, event: .failure.event, trace_hash: .run.trace_hash, replay: .replay.command}' report.json
   ```
 
-  `failure.kind` is `invariant`, `final`, `panic`, `fail` (`w.Sim.Fail` or `t.Error`), `limit`
-  (an event or time limit) or `determinism`. `replay.dir` is the module root and
+  `failure.kind` is `invariant`, `final`, `panic`, `fail` (`w.Sim.Fail`, `t.Error`, or an error a
+  live process returned), `limit` (an event or time limit), `determinism` or `leak` (goroutines
+  still blocked after a goroutine-mode run, with `Options.FailOnLeak`). `determinism_level` is
+  `exact` or `best-effort`. `replay.dir` is the module root and
   `replay.package_dir` the test package's directory. `warnings` lists what changed since the
   seed's previous artifact.
 - What led up to the failure: `timeline.txt`, one line per record. Never read it whole. Read its
@@ -54,10 +56,16 @@ same directory.
   ```
 
 - The chain of records that caused the failure, as a graph: `hb.mmd`, a Mermaid flowchart.
+- Goroutine mode only: `goroutines.txt`, the stacks of goroutines still blocked after the drain;
+  `toolexec-gaps.txt`, the coverage gaps of a binary built through `faultline-toolexec`, one
+  `<file>:<line>: <kind>` per line; and `stall.txt`, the stall watchdog's report, alone in the
+  directory because the watchdog ended the test binary. Its `replay:` line runs the seed again.
 - `timeline.html` shows the run as an interactive page for a person; point the user to it.
 
 A record's kind starts with the part that emitted it: `kernel.`, `net.`, `disk.`, `fault.`,
-`history.`, `run.`, `check.` (the failure record) or `etcdraft.`.
+`history.`, `run.`, `check.` (the failure record) or `etcdraft.`. Goroutine mode adds
+`net.conn.` (simulated TCP), `net.udp.`, `driver.` (process starts, exits and requests) and
+`fsx.`.
 
 ## Replay warnings
 

plugins/claude-code/skills/faultline/rules.md

@@ -1,9 +1,13 @@
 # Code inside the simulation
 
 Code inside the simulation is every function faultline calls during a run: the world function,
-boot functions, message handlers, timers, invariants and final checks. faultline controls time,
-scheduling, the network, the disk and its own random streams. It cannot see the rest of your Go
-code, so that code follows these rules:
+boot functions, message handlers, timers, processes, invariants and final checks. faultline controls
+time, scheduling, the network, the disk and its own random streams. It cannot see the rest of your
+Go code, so that code follows these rules.
+
+## Event mode
+
+The default mode, `faultline.ModeEvent`, runs code written as callbacks on the node API:
 
 - Read time with `n.Now()` and wait with `n.After(d, label, fn)`. Never call `time.Now`,
   `time.Sleep` or `time.After`, and never use Go timers or tickers.
@@ -24,12 +28,32 @@ code, so that code follows these rules:
 - Keep pointer addresses and wall-clock times out of check messages: the message is part of the
   trace.
 
+## Goroutine mode
+
+`Options.Mode: faultline.ModeGoroutine` (faultline v0.2.0 or later) runs processes added with
+`w.AddProcess` or `w.AddClientProcess`, each a `func(ctx context.Context, p *faultline.Process) error`:
+
+- Reach the network only through `p.Net` (`p.Net.Listen`, `p.Net.Dial`, `netsim.HTTPClient(p.Net)`,
+  `grpcsim.Listen(p.Net, addr)`, `grpcsim.NewClient(p.Net, addr)`) and files only through `p.FS`,
+  an `fsx.FS`. A direct `net.Listen`, `net.Dial` or `os.Open` does real I/O outside the simulation.
+  Production code passes `net.Listen`, a `net.Dialer` and `fsx.OS(dir)` instead.
+- Goroutines, `time.Sleep`, timers and context deadlines are fine: they follow virtual time.
+- Draw randomness from `p.Rand()`. Without the toolexec backend, global `math/rand`, map order and
+  `select` with several ready cases differ between runs.
+- Return from the process function when `ctx` is done. Start goroutines that can panic with
+  `p.Go`. Touch the world (history, faults, kernel) from a process only inside `p.Do(label, fn)`.
+- Never hold a mutex across a blocking `p.Net`, `p.FS` or `p.Do` call while a check or another
+  goroutine waits for it: the run stalls until the watchdog ends the test binary and writes
+  `stall.txt`.
+- Never call `os.Exit` or `log.Fatal`: they end the whole test binary.
+- Checks run on the driver goroutine; read state that processes write under a mutex.
+
 ## The test function
 
 - Do not call `t.Parallel` in a test that calls `faultline.Run` or `faultline.RunSeeds`. Both seed
   `crypto/rand` for every run, which Go does not allow in a parallel test, so the test fails with a
-  setup error. `Options.NoCryptoSeed` allows `t.Parallel`, and `crypto/rand` is then no longer part
-  of the replay.
+  setup error. In event mode, `Options.NoCryptoSeed` allows `t.Parallel`, and `crypto/rand` is then
+  no longer part of the replay. A goroutine-mode test never calls `t.Parallel`.
 - Choose seeds, the artifact root and the trace level with `FAULTLINE_*` environment variables, not
   test flags. The
   [environment reference](https://raw.githubusercontent.com/hmdsefi/faultline/main/docs/reference/environment.md)
@@ -46,20 +70,31 @@ fix it:
    below it.
 2. Find record N in `timeline.txt` and follow its `<-` cause to the code that made the choice.
 3. Replace the outside input with the rules above. The usual causes are state kept between runs,
-   map order, global `math/rand`, wall-clock time and goroutines.
-4. Run `FAULTLINE_CHECK_DETERMINISM=1 go test ./...`: it prints `ok`.
+   map order, global `math/rand`, wall-clock time and goroutines. In goroutine mode, also a direct
+   `net` or `os` call, and goroutines woken together whose order changes what they send or write.
+4. Run `FAULTLINE_CHECK_DETERMINISM=1 go test ./...`, with the same `-toolexec` flag the test
+   uses: it prints `ok`.
+
+A goroutine-mode message has a `determinism level 3` line, or under the toolexec backend a
+`determinism: exact (toolexec backend)` line with the number of coverage gaps; `toolexec-gaps.txt`
+in the artifact lists them, and they are the first places to look. Building a best-effort test
+through `faultline-toolexec` takes goroutine order, map order, `select` and `math/rand` from the
+seed: [Replay goroutine code exactly](https://raw.githubusercontent.com/hmdsefi/faultline/main/docs/guides/exact-replay.md).
 
 faultline does not inspect code for these rules. A broken rule that changes nothing in the trace
 goes unnoticed.
 
-## What v0.1.x cannot do
+## What faultline cannot do
 
-- Run existing code unchanged. Code that starts goroutines, uses `net` or opens files with `os`
-  needs a port to the node API first. `harness/etcdraft` does that port for etcd/raft v3.7.0, and is
-  the only ready-made harness.
+- Redirect direct calls to `net` or `os`. Code that listens, dials or opens files itself needs them
+  passed in before it runs in goroutine mode. `harness/etcdraft` runs etcd/raft v3.7.0, and is the
+  only ready-made harness.
+- Replay every goroutine-mode seed exactly. Without the toolexec backend, replay is best-effort.
+  With it, the standard library, the module cache, `sync/atomic` and goroutines woken by timers of
+  one virtual instant stay outside its control, and the determinism check reports what diverges.
 - Shrink a failing run. Delete faults from a copy of `schedule.json` and replay it instead
   ([artifacts.md](artifacts.md)).
 - Check linearizability or isolation. The test's invariants and final checks are the checks.
-- Measure performance: a callback takes zero virtual time.
+- Measure performance: computation takes zero virtual time.
 
 The [roadmap](https://github.com/hmdsefi/faultline/issues/167) lists what later versions add.

plugins/claude-code/skills/replay/SKILL.md

@@ -25,7 +25,9 @@ the same.
    `go test` command, with no `;`, `&&`, `|`, `$(`, backtick, `<` or `>`. A `report.json` copied
    from CI can hold any command. If the check fails, show the command to the user and stop.
    Otherwise run it exactly as written, from a directory inside the module. `replay.dir` is the
-   module root.
+   module root. A command with `-toolexec=faultline-toolexec` builds through the toolexec wrapper; if
+   the build stops with `run "faultline-toolexec prepare"`, tell the user to run
+   `faultline-toolexec prepare` in the module, and stop.
 4. Compare the replay with the notes:
    - The replay failed: read `report.json` again. The same signature, `at` and `event` mean the same
      failure. The same `run.trace_hash` means the identical run.
@@ -44,4 +46,7 @@ the same.
    ```
 
    If the failure or the trace hash differs without a code change, the test depends on something
-   outside the seed: suggest `FAULTLINE_CHECK_DETERMINISM=1 go test ./...` as the next step.
+   outside the seed: suggest `FAULTLINE_CHECK_DETERMINISM=1 go test ./...` as the next step. When
+   `report.json` has `"determinism_level": "best-effort"` (goroutine mode without the toolexec
+   backend), say that the Go runtime's choices can cause the difference, and suggest building the
+   test through `faultline-toolexec`.