feat(dataset-quality): count skills in trajectory coverage by omkargaikwad23 · Pull Request #568 · GoogleCloudPlatform/evalbench

omkargaikwad23 · 2026-08-12T09:11:26Z

trajectory_coverage only looked at MCP tools. So a product that ships skills got
graded against a surface its CUJs never name, and a skills-only product fell out of
the grade entirely.

The catalog is now every channel the product declares:

MCP tools and skill scripts, matched against expected_trajectory
skills themselves, matched against expected_skills

New generators/models/skills_catalog.py reads setup.skills — clones a git URL or
scans a local path, and takes the skill name from SKILL.md frontmatter. An entry
that resolves to nothing raises instead of being skipped, since a short catalog
quietly inflates coverage.

Also renamed the sub-scorer metric keys — dropped the dq_ prefix and fixed names
that no longer matched what they count (total_tools → capabilities_total). They
live inside a JSON blob, so nothing downstream changes.

Test plan

pytest test/dataset_quality_test.py -q — 88 passed
Live Cloud SQL run: 63 (C), graded on 100/100 weight, no excluded scorers
Tested all internal MCP servers and skills agaist available CUJs
Config declaring both channels: coverage went 0/15 → 5/63

…uality evaluation

The bump to 30 came from an earlier design with two activation scorers sharing a 30 budget, which we did not build. Covering more channels makes the measurement more complete, not more important than error recovery or composition, and upweighting the lowest-scoring metric lowers every dataset's grade for no stated reason. Restores the registry budget to 100.

omkargaikwad23 · 2026-08-13T04:23:17Z

/gcbrun

…y and clarity

…d tools and skills

…ine descriptions in trajectory coverage scorer

prernakakkar-google · 2026-08-13T08:28:22Z

The denominator quadruples (0/15 → 5/63 in your own test), so dataset_quality scores for skills-bearing products aren't comparable across this change. What's the plan for the step change on the trends page — new series, or annotate the boundary? Same question for the dq_* rename: code is fine (the viewer renders metrics generically), but historical BigQuery rows keep the old keys, so any saved query over the blob breaks at the boundary.

prernakakkar-google · 2026-08-13T08:23:54Z

+    raise SkillCatalogError(f"clone failed for {url}: {last_error}")
+
+
+def _scan_dir(root: str, wanted: set[str] | None = None) -> list[Skill]:


This raises only when zero wanted names match. With skills: ["alpha", "typo", "gamma"] against a root holding alpha/beta/gamma, it returns 2 and stays silent — which is exactly the shrunken denominator the module docstring says it prevents. Can we raise on any unmatched name in wanted, and extend test_a_narrowing_key_matching_nothing_is_fatal to the partial-miss case?

Good catch. Any unmatched name now raises with all the missing ones listed, and added a test for the partial-miss case.

prernakakkar-google · 2026-08-13T08:25:50Z

-                ),
-            )]
+            tools = self._fetch_tools(setup)
+            skills = resolve_skills(setup)


A skills-resolution failure now nulls the entire report, including the five judge sub-scorers that never touch skills. gemini_cli_skills_model.yaml:12 hardcodes an absolute path under one person's home dir — on any other machine that config goes from "grades the tools" to "ungraded." Should a skills failure degrade skill coverage rather than the whole grade?

There's no separate skills sub-scorer to degrade, trajectory_coverage grades tools, scripts and skills as one union catalog, so a skills failure would shrink denominator and inflate the score rather lower it.
On gemini_cli_skills_model.yaml: that absolute path is worth fixing but it's separate file config change.

Agreed that silently shrinking the catalog is worse than failing. But the degradable unit is trajectory_coverage, not a separate skills scorer: _run_scorers already catches a sub-scorer exception, marks it applicable=False, and grading.py reports it in excluded_scorers with the remaining weight. Moving resolve_skills into TrajectoryCoverageScorer.run would drop just that scorer's 20 weight on a bad catalog and still grade the five judges — no inflated denominator, no lost report. As it stands a bad setup.skills path nulls vague_examples, cuj_diversity, error_recovery, composition, and parameter_coverage, none of which touch the catalog.

Good call. Split the try. Tool discovery still nulls the report since every sub-scorer reads tool_names, but a SkillCatalogError now just marks trajectory_coverage inapplicable, so it lands in excluded_scorers and the other 80 weight grades as normal. Added a test asserting exactly that.

omkargaikwad23 · 2026-08-13T10:09:15Z

The denominator quadruples (0/15 → 5/63 in your own test), so dataset_quality scores for skills-bearing products aren't comparable across this change. What's the plan for the step change on the trends page — new series, or annotate the boundary? Same question for the dq_* rename: code is fine (the viewer renders metrics generically), but historical BigQuery rows keep the old keys, so any saved query over the blob breaks at the boundary.

Thanks for the review @prernakakkar-google!
Only 2 products declare skills today, and one each, so the denominator barely moves for them. Tools-only products are unaffected -> same catalog, same score.

For the dq_ rename, nothing consumes those keys by name; the viewer renders whatever the blob holds.

…te related documentation and versioning.

prernakakkar-google · 2026-08-14T09:30:55Z

-# standard agent pipeline: a no-op generator does nothing, and the `dataset_quality`
-# scorer grades the whole dataset holistically (each sub-scorer computes a 0-100
-# sub-score; grading rolls them into a weighted global score + letter grade).
+# Grades a product's CUJ configs, not a model: no agent runs and nothing is


Can we simplify this to say It helps with Datset CUJ grading.

Makes sense!

prernakakkar-google · 2026-08-14T09:34:39Z

+    stem, suffix = os.path.splitext(filename)
+    if filename.startswith(".") or filename == "__init__.py":
+        return False
+    if suffix.lower() in _NON_SCRIPT_SUFFIXES:


I think it would be more accurate to check in given list of .js, .ts, .py, .sh

prernakakkar-google · 2026-08-14T11:46:45Z

-                ),
-            )]
+            tools = self._fetch_tools(setup)
+            skills = resolve_skills(setup)


Agreed that silently shrinking the catalog is worse than failing. But the degradable unit is trajectory_coverage, not a separate skills scorer: _run_scorers already catches a sub-scorer exception, marks it applicable=False, and grading.py reports it in excluded_scorers with the remaining weight. Moving resolve_skills into TrajectoryCoverageScorer.run would drop just that scorer's 20 weight on a bad catalog and still grade the five judges — no inflated denominator, no lost report. As it stands a bad setup.skills path nulls vague_examples, cuj_diversity, error_recovery, composition, and parameter_coverage, none of which touch the catalog.

prernakakkar-google · 2026-08-14T11:47:39Z

Few minor comments, otherwise LGTM

…d of failing the entire dataset evaluation

feat: implement skill catalog resolution and integrate into dataset q…

55505a1

…uality evaluation

omkargaikwad23 requested review from IsmailMehdi, helloeve and prernakakkar-google as code owners August 12, 2026 09:11

omkargaikwad23 added 5 commits August 13, 2026 05:14

refactor: rename dataset quality metrics to improve naming consistenc…

c218ec4

…y and clarity

Merge remote-tracking branch 'origin/main' into dq-skills-coverage

804a25c

refactor: inline skill resolution logic and add logging for discovere…

4ab3e89

…d tools and skills

refactor: move _describe to a standalone utility and sanitize multi-l…

718b287

…ine descriptions in trajectory coverage scorer

Merge remote-tracking branch 'origin/main' into dq-skills-coverage

b160030

prernakakkar-google requested changes Aug 13, 2026

View reviewed changes

refactor: improve skill catalog matching and error handling, and upda…

97586d7

…te related documentation and versioning.

omkargaikwad23 requested a review from prernakakkar-google August 13, 2026 10:10

omkargaikwad23 and others added 3 commits August 14, 2026 06:22

Merge branch 'main' into dq-skills-coverage

f96d175

Merge branch 'main' into dq-skills-coverage

f2110c8

Merge branch 'main' into dq-skills-coverage

6d24ca8

prernakakkar-google previously approved these changes Aug 14, 2026

View reviewed changes

feat: handle unresolvable skills by skipping dependent scorers instea…

fe18baf

…d of failing the entire dataset evaluation

omkargaikwad23 dismissed prernakakkar-google’s stale review via fe18baf August 14, 2026 13:49

omkargaikwad23 requested a review from prernakakkar-google August 14, 2026 13:50

helloeve approved these changes Aug 14, 2026

View reviewed changes

omkargaikwad23 merged commit 2768d7c into main Aug 14, 2026
11 checks passed

release-please Bot mentioned this pull request Aug 14, 2026

chore(main): release 1.16.0 #571

Open

		raise SkillCatalogError(f"clone failed for {url}: {last_error}")


		def _scan_dir(root: str, wanted: set[str] \| None = None) -> list[Skill]:

Uh oh!

Conversation

omkargaikwad23 commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test plan

Uh oh!

omkargaikwad23 commented Aug 13, 2026

Uh oh!

prernakakkar-google commented Aug 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omkargaikwad23 Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omkargaikwad23 commented Aug 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prernakakkar-google commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

omkargaikwad23 commented Aug 12, 2026 •

edited

Loading

omkargaikwad23 Aug 13, 2026 •

edited

Loading