Repository navigation
evals: add 13 new evals across 4 skills - #11
Merged
Merged
Conversation
…up app-package Adds 13 new evals across four eval files: - data/evals_feed-operations.json (4 evals, new): bulk-feed-jsonl, fix-broken-feed-file, conditional-write-and-tensor-update, product-partial-updates (lab-inspired). Exercises JSONL wire format, doc-id format, partial-update operator wrapping, weightedset vs array add shapes, conditional writes, and tensor modify cells. - data/evals_pyvespa.json (3 evals, new): build-app-package-script, feed-and-query-script, fix-broken-pyvespa-script. Covers ApplicationPackage construction, indexing tuple-vs-list gotcha, app name rules, attribute=list signature, feed_async_iterable, hybrid query with input.query(q). - data/evals_query-builder.json (3 evals, new): ecommerce-multi-signal- rank-profile, hybrid-rrf-global-phase, fix-broken-yql. Exercises multi-signal first-phase, global-phase reciprocal_rank_fusion, match-features for debug, weakAnd annotation syntax, grouping pipe+all wrapper. - data/evals_app-package.json (+2 evals): add-query-profile-with-tensor- type, deployment-xml-multi-region-with-tests. Covers query-profile type linking with XML-escaped tensor types and Cloud deployment.xml. Fixtures live under data/fixtures/<eval-name>/ for diagnose-and-fix evals. Lab-inspired evals (product-partial-updates, ecommerce-multi- signal-rank-profile, hybrid-rrf-global-phase) draw on the e-commerce patterns from learn_git's labs, but ship self-contained fixtures so the repo does not depend on external paths.
…loyment.xml
Cross-checked technical claims in the new evals against
/Users/abhishek/workspace/documentation/en/ and the pyvespa source at
/Users/abhishek/workspace/pyvespa. Two real errors fixed:
- feed-operations eval 3 (conditional-write-and-tensor-update): the
tensor `modify` long form uses key `"addresses"` (plural), not
`"cells"`. `"cells"` is the documented short form for tensors with a
single mapped dimension (a map of key:value). The prompt now steers
the agent to the short form for tensor<float>(topic{}); assertions
accept either short or long form via regex alternation.
- app-package eval 5 (renamed deployment-xml-multi-region-with-tests
-> deployment-xml-multi-region-rollout): dropped the bogus <test/>
and <staging/> self-closing siblings — neither form is in the
documented deployment.xml schema. The eval now exercises the real
documented constructs: <region>, <delay minutes="N"/>, <parallel>,
and <block-change days hours time-zone/>, matching the canonical
staged-rollout-with-parallel-fan-out pattern in the deployment
reference.
- pyvespa eval 1 (build-app-package-script): removed file_exists
assertions for ./output/services.xml and ./output/schemas/article.sd
— these required actual script execution which depends on pyvespa
being installed in the agent's environment, which is not guaranteed.
Script correctness is now graded statically.
Verifications performed (all grounded):
- Doc ID double-colon form
- weightedset add object form {"key": weight}
- conditional write "condition" at top-level with <type>.<field> selection
- remove takes only doc ID, no fields block
- tensor modify operations: replace/add/multiply
- grouping | all(group(...) each(...)) wrapper
- YQL annotations use colon: {targetHits: 200}
- rank-profile inherits, match-features, global-phase, rerank-count
- reciprocal_rank_fusion is a documented pseudo-function
- Rank features: bm25, nativeRank, freshness, closeness, attribute
- pyvespa: name rules enforced at __init__ (lowercase a-z0-9, letter-
start, <=20 chars, no hyphens/underscores)
- pyvespa: Field attribute is Optional[List[str]]; indexing tuple
renders multiline, list renders pipe-separated
- pyvespa: is_successful() is canonical, is_successfull() is deprecated
- services.xml: document mode required, valid values index/streaming/
store-only
- redundancy and min-redundancy both work, min-redundancy preferred
- query-profile-type field name ranking.features.query(NAME) with
XML-entity-escaped tensor type
- deployment.xml: <region>, <delay>, <parallel>, <block-change>
- Vespa Cloud regions: aws-us-east-1c, aws-eu-west-1a, gcp-us-central1-f
abhishekkrthakur
marked this pull request as draft
May 18, 2026 07:25
Five assertions failed on legitimate agent output:
- pyvespa indexing tuple: required single-element single-line tuple; agents legitimately use multi-line and multi-element tuple forms which still render correctly per pyvespa's Field.indexing_to_text
- pyvespa first_phase: required direct string after =; agents legitimately wrap in parens first_phase=("...")
- pyvespa ranking: required body-dict form "ranking":"hybrid"; sess.query accepts kwarg form ranking="hybrid"
- query-builder category filter: required = operator; agent legitimately used contains, both valid in YQL on attribute fields
After this, every assertion passes on real output except one legitimate skill-uplift signal: fix-broken-pyvespa-script without_skill misses the list-vs-tuple gotcha, which is the eval's intended discrimination.
Collaborator
Author
Local harness resultsRan all 13 new evals locally against Claude with both Sonnet (default) and Haiku to validate the evals and measure the skill-vs-no-skill delta. Sonnet (iteration 1)
Haiku (iteration 2)
Key findings
Bonus: 4 brittle regex assertions fixed (commit a122787)The first run revealed legitimate alternative output that my regexes didn't account for:
All four were the agent doing something valid that my pattern was too strict to accept. |
abhishekkrthakur
marked this pull request as ready for review
May 18, 2026 07:52
thomasht86
reviewed
May 18, 2026
Co-authored-by: Thomas Hjelde Thoresen <thomas.h.thoresen@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds eval coverage for the four skills that had zero or thin coverage. After this PR, every skill except vespa-cli has ≥3 evals.
What's added
data/evals_feed-operations.json(4 evals, new)bulk-feed-jsonl— generate a JSONL feed with PUT/UPDATE/REMOVE, partial-update operators, weightedset vs array add shapesfix-broken-feed-file(diagnose) — single-colon doc ID, bare value in update, remove with extraneous fields, wrong weightedset add shapeconditional-write-and-tensor-update— top-levelcondition, mapped-tensormodifyoperator, atomic multi-field updatesproduct-partial-updates(lab-inspired) — realistic e-commerce partial updates against a providedproduct.sdfixturedata/evals_pyvespa.json(3 evals, new)build-app-package-script—ApplicationPackage+Field/HNSW/RankProfile, exercises the tuple-vs-list indexing gotcha and attribute must be a list signaturefeed-and-query-script—feed_async_iterable+app.syncio()+ hybrid YQL withinput.query(q)fix-broken-pyvespa-script(diagnose) — app name with hyphen,attribute="fast-search"bare string, embed-indexing as list instead of tupledata/evals_query-builder.json(3 evals, new)ecommerce-multi-signal-rank-profile(lab-inspired) — append three rank profiles to a fixture schema combining BM25, rating, in-stock bonus, freshness, with match-featureshybrid-rrf-global-phase—inherits semantic, function reuse,global-phasewithreciprocal_rank_fusion+ rerank-countfix-broken-yql(diagnose) —{targetHits=200}(should be:), grouping missing the| all(...)wrapperdata/evals_app-package.json(+2 evals)add-query-profile-with-tensor-type— addsdefault.xmllinked viatype="root"andtypes/root.xmlwith XML-escaped tensor typedeployment-xml-multi-region-with-tests— Cloud<deployment>with<test/>/<staging/>before<prod>, three regions in a single prod blockFixture & lab strategy
Lab-inspired evals (
product-partial-updates,ecommerce-multi-signal-rank-profile,hybrid-rrf-global-phase) draw on the e-commerce patterns fromlearn_git's labs anduniversity/101/chN/ecommerce_app/reference solutions, but ship self-contained fixtures underdata/fixtures/<eval-name>/so the repo has no external-path dependencies.Workflow
Also updates
.github/workflows/run-harness.ymlto include the three new eval files in the matrix so they actually run when the harness is dispatched.