fix(scanner): keep a turn's final tallies when a scan lands mid-stream

[NickAme03]

What

A turn stored while its response was still streaming never gets corrected. It stays frozen at the partial token counts for the life of the database.

The same transcript gives two different totals depending on when you scanned it:

	output	cache_creation
one scan, after the response finished	900	3000
a scan that landed mid-stream	40	3000
...after a second scan reads the final record	40	3000

Why

Dedup happens at two levels, and only the first one is right.

In memory, per file, seen_messages[message_id] = turn (scanner.py:441, and again at :764 on the incremental path). Last record wins, which is correct: on my local corpus, of the 2678 message_ids with more than one record, all 2678 have the last record equal to the field-wise maximum.

In the database, the unique index on message_id (scanner.py:127-130) plus INSERT OR IGNORE (scanner.py:561). First write wins, permanently.

So a scan that starts mid-stream sees only the partial snapshots written so far, stores those, and every later scan re-reads the final record and drops it on the index. Nothing repairs it afterwards: the end-of-scan recompute (scanner.py:801-808) re-sums from the frozen turns, and the Rescan button says in its own tooltip that it adds new turns "without affecting existing history".

The window is wider than it sounds. Between the first and last record of one message_id, my corpus shows a median of 3.1s, p90 of 17.9s, max of 246s. 73% of multi-record messages span at least a second.

Impact: small, and I'd rather say so

Roughly 1-2% of output tokens in the worst case I measured, and zero on the cache and input fields, which already arrive complete in the first record.

Scans today only run at dashboard startup (cli.py:423), on the Rescan button, and from the CLI, so in practice this bites the message that happened to be streaming when you opened the dashboard. Simulating a periodic scan against the real timestamps in my corpus, averaged over 12 phase offsets:

scan interval	messages frozen	output lost
every 30s	0.94%	2.42%
every 60s	0.62%	1.97%
every 300s	0.25%	0.55%

I'm not going to dress this up as a big number, because it isn't one. The reason to fix it anyway is determinism: the same bytes on disk should give the same total whenever you scan them, and right now they don't. The wrong value is also silent and permanent, which is the part that bothered me.

Worth noting that PR 157 and PR 161 both make scanning periodic, and neither touches this path, so whichever of them lands moves this from the 300s row of that table to the 30s row.

Change

INSERT OR IGNORE becomes an upsert that wins on the higher output_tokens. Same selection rule the in-memory level already implements, applied where it was missing.

The row is updated in place and never deleted, so the append-only history stays intact, including sessions whose transcripts Claude Code has since pruned.

Testing

• Full suite: 148 passed (147 before, plus the regression test).
• The new test fails without the fix: AssertionError: 40 != 900.
• Existing databases: DELETE FROM processed_files plus a rescan repairs frozen turns (40 to 900) and preserves sessions whose transcript files no longer exist. Two turns before, two after, nothing dropped. Happy to add that as a one-time migration if you want it, I left it out to keep the diff small.

One thing I could not check locally: I'm on SQLite 3.50.4 / Python 3.14. Upsert with a conflict target on a partial index needs SQLite 3.24+ (June 2018), so the 3.9 / 3.11 / 3.12 CI matrix should settle it.

Unrelated, noticed while reading: AGENTS.md:85 still describes POST /api/rescan as deleting the DB and running a full rescan, which the current handler explicitly does not do. Left alone here, happy to file it separately.