feat(#126): trace research/dev runs and record code-gen model, usage and cost

[alexosugo]

Description

Top of the #127 → #162 → #163 stack. It is rebased on the approved #162 (Langfuse v4 migration), and its diff here is the nine commits on top of #162.

Research and development tracing

• Research and development supervisor runs are traced with per-step observations. Before this, only the memory pipeline was traced.

Code-gen generations in Langfuse

• Each LLM call now appears under generate-code, carrying the model that actually ran, token usage (including cache tokens), and cost where the path reports it.
  • claude-code-cli: code-gen-plan, code-gen-execute, code-gen-execute-relaxed
  • claude-api: code-gen-plan, code-gen-file, code-gen-continuation
• A CLI is_error marks the generation, and the trace, as ERROR.
• A paid call that fails keeps its model, usage and cost on the ERROR generation. This covers a CLI is_error, CLI or API output that is not valid JSON, and JSON that fails the triage/distill schema.
• The new observeActiveGeneration helper attaches a generation to whatever step is running and records nothing outside a trace.

CLI output parsing (shared src/llm/cli-envelope.ts)

• Handles both output shapes: the single result object, and the JSON array of every message that the CLI prints with --verbose or the verbose setting.
• Takes the last result message, and uses the errors list as the result text for error results such as error_max_turns.
• Reports the most expensive model in modelUsage and sums usage across all models.
• Still finds the result message when warnings or other noise around the JSON contain brackets.

Cost on the code-gen result

• modelUsed, tokensUsed and costUsd (new on CodeGenModuleOutput) are passed from the module through CodeGenerationAgent.
• They show up in the generate-code output and on a new console summary line.
• For the CLI they are totals across plan, execute and relaxed retry, attributed to the most expensive model.

Verification

• Local: 1171 tests passing; typecheck and lint clean.
• Live Langfuse check of one CLI generation (it used the module's CLI-call code copied into a script): nested under generate-code, claude-opus-5-5, 81.2k tokens, $0.56, included in the trace total.
• No end-to-end pipeline run.

Known gaps

• If generation throws partway through, the spend is missing from the module result. It still appears on the individual generations.
• A run that ends one turn short of the max-turns cap without is_error is marked partial on the module result, and its generation is not ERROR. A real max-turns stop (error_max_turns, is_error: true) is ERROR.
• opencode is still a stub with no LLM calls to trace.
• Finding the result envelope is still quadratic in the number of candidate lines: 1,200 noisy lines that look like JSON took over 5 s in a local check. This is left for a follow-up.

#126

Code review checklist

• Readable: Concise, well named, follows the style guide, documented if necessary.
• Tested: Unit and/or e2e where appropriate
• Backwards compatible: Works with existing data and configuration or includes a migration. Any breaking changes documented in the release notes.

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

[sugat009]

Thank you. The tracing and the shared CLI parser work well. No item blocks the merge.

Please merge #162 into this branch after #162 takes its suggestions. The #162 docs fix also corrects docs/observability.md:238-239 here.

Known gaps in the description:

• Gap 2 is not correct for CLI 2.1.281. A max-turns stop is ERROR. See cli-envelope.ts:91.
• Please add one gap: a CLI is_error on the provider path records no spend. See claude-cli.ts:318.

CI is green. With all the suggestions applied, 1050 tests pass locally.

[sugat009]

Thank you. All 7 round-1 points are fixed. The failed-call spend is now fixed in code, not only listed as a gap. I checked each point on b4c08a0. Local gates pass with 1055 tests. The CI test job did not run on this head.

Please merge feat/langfuse-v4-migration (012d77e) into this branch, and then I will approve. My trial merge conflicts in two places only:

• docs/observability.md: take #162's Delivery Semantics paragraph, and keep your warning-label sentence.
• test/observability/index.spec.ts: keep both new tests.

The merged tree passes all gates with 1057 tests. Also, the PR description still says "Handles the newer JSON array output" and "1047 tests passing".

The inline comments are non-blocking.

[alexosugo]

@sugat009 You can review this just this PR at the top of the stack. It contains all the commits from the downstream PRs and is representative of the entire state. Folding #162 into #163 would require unstacking.

[sugat009]

Approved. Thanks for the rebase. It covers my request: #163 now holds #162's head, so there is nothing to unstack. I reviewed #163 as the whole stack. All my comments are resolved, and CI is green.

suggestion (non-blocking): .env.example:66-67 sets the placeholder keys pk-lf-... and sk-lf-.... A developer who copies the file turns tracing on with invalid keys, and each local run then sends its prompts to cloud.langfuse.com. Please comment out the two lines, and change the comment above them to say that tracing stays off until both keys are set. These lines come from #127, so a follow-up is fine.