feat(eval): golden-prompt harness over frozen review inputs - #186
Open
alansikora wants to merge 2 commits into
Open
alansikora wants to merge 2 commits into
alansikora wants to merge 2 commits into
Conversation
Prompt edits are invisible in code review. Their effect surfaces later as a change in the findings, by which point the cause is hard to attribute — and instruction overhead is paid on every review forever, so growth that nobody measured compounds silently. This adds the layer that makes prompt construction observable: - internal/evalcorpus — the on-disk fixture format. A fixture is everything a review needs to build its prompt (PR metadata, diff, the file contents the reviewer read, project docs, config), so rendering one touches neither the network nor a working tree. It declares its own structs rather than reusing review's types, which keeps it dependency-free (no import cycle for review's own tests) and pins the serialised format against refactors inside review. - cmd/evalsnap — captures a fixture by reusing review's real fetch path, so a fixture is what a review would have seen rather than an approximation. Not part of the shipped binary. - TestPromptGolden — renders every fixture and diffs against a checked-in golden, reporting the first differing line and the size delta rather than dumping two multi-kilobyte prompts. SIZES.txt records each prompt's rendered size so growth lands in the diff. Validated against the open prompt chain (#176-#180): the harness reports +4,599 characters, identical across all three fixtures, and names the section responsible. This measures prompt construction, not review quality — no model runs. Judging whether a prompt change helps needs labelled findings, repeated runs to establish variance, and real model calls. That layer does not exist yet, and the variance measurement has to come first: with ~5 findings per PR, run-to-run noise can swallow the effect being looked for. Corpus fixtures embed full file contents, so a fixture from a private repository must never be committed to this public one. evalsnap checks the source repo's visibility with GitHub and refuses to write into any git-tracked directory, before it fetches or reads anything.
Deploying codecanary with
|
| Latest commit: |
e193347
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://8aff4e07.codecanary.pages.dev |
| Branch Preview URL: | https://feat-prompt-eval-harness.codecanary.pages.dev |
![codecanary-bot[bot]](https://avatars.githubusercontent.com/in/3176464?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
A deleted or never-committed SIZES.txt made writeSizeReport return without reporting anything, so the check passed silently — defeating the one thing the file is for. Distinguish a missing report from a matching one and fail on the former.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Torna visível o que hoje é invisível: mudanças na construção do prompt.
Por quê
Edição de prompt não aparece em code review. O efeito surge depois, como mudança nos findings, quando já é difícil atribuir a causa. E overhead de instrução é pago em toda review, para sempre — então crescimento que ninguém mediu vai compondo em silêncio.
Concretamente: a cadeia aberta #176–#180 adiciona +4.599 caracteres ao prompt base, idêntico em todos os fixtures. Isso não estava escrito em lugar nenhum. Com este harness, estaria — no diff.
O que entra
internal/evalcorpus— formato do fixture em disco. Um fixture é tudo que uma review precisa para montar o prompt (metadados do PR, diff, conteúdo dos arquivos que o reviewer leu, project docs, config), então renderizar não toca rede nem working tree. Declara structs próprias em vez de reusar os tipos dereview: mantém o pacote sem dependências (evita ciclo de import nos testes do próprioreview) e fixa o formato serializado contra refactors internos.cmd/evalsnap— captura um fixture reusando o caminho real de fetch doreview(mesmoFetchPR, mesmo leitor de conteúdo com os mesmos filtros de tamanho e ignore, mesma descoberta de docs). O fixture é o que uma review teria visto, não uma aproximação. Não faz parte do binário shipado —go build ./cmd/reviewnão puxa.TestPromptGolden— renderiza cada fixture e compara com golden commitado. Reporta a primeira linha divergente e o delta de tamanho, em vez de despejar dois prompts de vários KB.SIZES.txtregistra o tamanho de cada prompt.O que isso NÃO faz
Não mede qualidade de review. Nenhum modelo roda, nada é julgado.
Saber se uma mudança de prompt melhora ou piora exige findings rotulados, rodadas repetidas para estabelecer variância, e chamadas reais de modelo. Essa camada não existe ainda — e a medição de variância tem que vir primeiro: com ~5 findings por PR, o ruído entre rodadas pode engolir inteiro o efeito que se quer medir.
Corpus e repositórios privados
Um fixture embute o conteúdo completo dos arquivos. Fixture de repositório privado nunca pode ser commitado neste repo público — o histórico do git torna o vazamento permanente.
evalsnapconsulta a visibilidade do repo no GitHub e se recusa a gravar em qualquer diretório rastreado pelo git, antes de buscar ou ler qualquer coisa. Testado nos três casos (privado+rastreado recusa, privado+fora de git passa, público+rastreado permite) egitWouldTracktem teste unitário cobrindo as bordas.Corpora maiores, de repos privados, vivem fora daqui via
$CODECANARY_EVAL_CORPUS. Os 3 fixtures commitados (560KB, dos PRs públicos deste repo) existem para o harness rodar em CI e para a forma ficar à vista de quem clona — não são um conjunto de avaliação sério.go build,go vet,go test ./...egolangci-lint run ./...limpos.