fix: keep --no-llm negative cases off-skill by mimran-khan · Pull Request #100 · NVIDIA/SkillEvaluator

mimran-khan · 2026-08-26T20:07:02Z

Summary

--full --no-llm was generating a negative case that asked what the skill does by name. That is an explicit invocation, not a negative.

The default prompt is now an unrelated question, expected_skill stays null, and the behaviors talk about not applying this skill. Fixes #90.

Verification

I am familiar with the Contributing Guidelines
Added or updated focused tests
Updated documentation for user-visible changes
Ran make lint
Ran make test
Ran make build
Did not add credentials, private datasets, or proprietary benchmark content

Release Impact

Updated CHANGELOG.md

The default template asked what the skill does by name, which is an explicit invocation, not a negative case. Use an unrelated prompt instead, and keep expected_skill null. Fixes NVIDIA#90 Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

rng1995

The focused generator suite passes (18 tests), and Ruff/diff checks are clean, but the deterministic negative remains invalid for a class of skills; reproduction is inline. CI also has a packaging smoke-test failure after build/install, in addition to the unrelated-history Gitleaks failure.

A hard-coded errand question is itself a positive for city/task skills. Choose the first canned prompt that does not share domain tokens with the skill name or description, and omit the negative when none is safe. Fixes NVIDIA#90 Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

# Conflicts: # CHANGELOG.md

mimran-khan · 2026-08-30T07:16:41Z

Merged main. The off-skill negative prompt fix is still in place. Ready for re-review.

Drop errand-themed canned negatives, match skill name substrings and token stems before accepting a template negative, and omit the negative bucket when no safe off-domain prompt exists.

mimran-khan · 2026-09-01T19:27:43Z

Merged latest main and pushed another round on the off-skill negative selection. Ready for re-review.

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

Merge main and use author-provided negative prompts from eval guidance when available. Planning-style skills no longer receive guessed canned negatives; the negative bucket is omitted unless a safe off-skill prompt is available. Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>

Resolve CHANGELOG.md after merging upstream NVIDIA#99, NVIDIA#107, and NVIDIA#112.

rng1995

The planning-specific case is improved, but the core safety criterion is still not met: an exact-head synonym-based media transcoder reproduction is assigned the canned WAV-to-FLAC question as an off-skill negative. The focused suite passes (22 tests), but Ruff also fails with SIM103, matching the red Python 3.12 CI job. Please remove the semantic false-negative path and clear both lint and DCO before re-review.

Merge main and treat media/transcode skills like planning: skip guessed negatives unless eval guidance supplies one. Simplify domain overlap check for Ruff SIM103. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

mimran-khan · 2026-09-12T15:34:45Z

Merged latest main (including #104) at 20ce6bb. On the negative-case side I tightened things again: planning and audio/media skills skip canned negatives unless eval guidance supplies one (covers the media-transcoder case), and the domain overlap helper is SIM103-clean. Ready for another look when you have a minute.

rng1995 · 2026-09-13T03:41:16Z

@mimran-khan - Please resolve merge conflicts

rng1995 · 2026-09-13T04:48:26Z

@mimran-khan, the remaining blockers at 20ce6bb are:

The canned-negative heuristic still produces semantic false negatives for synonym-only skills. For example, music-reencoder described as “Changes songs between codecs while keeping tags” still receives the WAV-to-FLAC prompt as an off-skill negative. Please use an explicitly authored negative, or omit canned negatives when the off-skill relationship cannot be established safely.
The branch currently conflicts with main; please update it while retaining both changelog entries.
DCO fails because commit 7b744e2 is missing a Signed-off-by trailer.

Please address these items and ping me when the PR is ready for re-review.

Drop canned negative prompts and vocabulary heuristics that could not reliably establish an off-skill relationship. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

mimran-khan · 2026-09-14T09:52:24Z

Head is still 3527105 (merged with current main). No-llm negatives are author-only now at 24232e6 / merge 3527105. Could you take another look when CI is approved on the fork?

mimran-khan · 2026-09-14T09:53:15Z

@rng1995 when you have a moment, could you re-review from the current head? I do not have permission to re-request reviewers on the upstream repo from my account.

rng1995 · 2026-09-14T16:03:28Z

-            ],
-        },
    ]
+    negative_question = _template_negative_question(skill, eval_hints)


[P1] @mimran-khan Keep the --full contract consistent when the negative bucket is omitted. This path now returns three cases without authored negative guidance, but the CLI still prints Mode: 4-bucket, its help, module documentation, README, and user documentation promise exactly four cases, and test_main_reports_created_dataset_with_written_payload[extra_args1-4] fails with 3 versus 4 on this head, matching the red Python CI. Please update the case-count contract, CLI wording, documentation, and existing regression consistently, or preserve the four-case contract with a demonstrably safe alternative. DCO also remains red for unsigned commit 7b744e2.

@rng1995 head 1c519713 has the --full contract/docs/tests aligned (up to four cases, three in template mode without an authored negative). CI and DCO are green. Ready for re-review.

Document and test that no-llm full mode emits three cases without an authored negative and four when eval guidance supplies one. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

rng1995 · 2026-09-16T03:41:00Z

-`--full` generates four case buckets — explicit, implicit, and contextual
+`--full` generates the explicit, implicit, and contextual buckets, plus a negative
+bucket when eval guidance supplies a safe off-skill prompt (template mode omits
+negative otherwise)


[P2] Remove the stale negative-case clause. This edit runs directly into the old “positives plus a negative case” text on the next line, so the paragraph now says template mode without guidance both omits and includes a negative, and the sentence has no punctuation. Please rewrite this as one consistent statement that distinguishes LLM mode from template mode.

Tier3 live eval section now matches the template vs LLM case counts (no more contradictory up-to-four wording in that doc).

rng1995 · 2026-09-16T03:41:01Z

+`--full` targets one case per bucket when a safe negative prompt is available.
+The mix matters: the negative case protects against a skill that triggers when
+it should not. In `--no-llm` template mode, the negative bucket is omitted
+unless eval guidance (for example `evals/EVAL.md`) includes an off-skill prompt.


[P2] Document the heading required for an authored negative. The parser only puts bullets into negatives under a negative/off-skill/counterexample heading, but the guidance section immediately below still says three headings are parsed and shows none for negatives. A user following these docs cannot produce the advertised fourth template bucket; please add ## Negative Cases (or another supported heading) and an example.

eval-datasets.mdx now calls out the ## Negative Cases section for authored negatives in template mode.

rng1995 · 2026-09-16T03:41:03Z

+    parser.add_argument(
+        "--full",
+        action="store_true",
+        help="Generate the full bucket set (up to 4 cases; negative only with authored off-skill guidance)",


[P2] Scope the authored-negative qualifier to template mode. Without --no-llm, this command calls _generate_with_llm(..., full=True), which still generates four buckets without authored negative guidance; this help and the mode text at line 1100 currently say otherwise. Please condition the wording on args.no_llm or explicitly say “in template mode.”

@rng1995 pushed 99ebb796: rewrote tier3-live-evaluation copy, documented ## Negative Cases in eval-datasets, and scoped generate_dataset --full help/mode text to template vs LLM. Ready for another look.

Distinguish four-bucket LLM generation from template --no-llm behavior, document EVAL.md negative headings, and scope CLI mode text to no-llm. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

Resolve CHANGELOG conflict after catalog fleet and parallel workers landed. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

mimran-khan requested review from chrisknvidia, mosheabr, rmalani-nv and rng1995 as code owners August 26, 2026 20:07

rng1995 requested changes Aug 28, 2026

View reviewed changes

Comment thread src/skillevaluator/tier3/generate_dataset.py Outdated

mimran-khan added 3 commits August 28, 2026 14:04

Merge origin/main into fix/no-llm-negative-case

e6ad00e

merge main into fix/no-llm-negative-case

6cafa62

# Conflicts: # CHANGELOG.md

mimran-khan added 3 commits August 31, 2026 22:34

fix: keep --no-llm negatives off planning and errand domains

7860f30

Drop errand-themed canned negatives, match skill name substrings and token stems before accepting a template negative, and omit the negative bucket when no safe off-domain prompt exists.

merge main into fix/no-llm-negative-case

f75145c

merge main into fix/no-llm-negative-case

ca2939c

mimran-khan added 5 commits September 3, 2026 15:52

merge main into fix/no-llm-negative-case

a018056

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

merge main into fix/no-llm-negative-case

b324add

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

chore: merge main into fix/no-llm-negative-case

0c46f3a

Resolve CHANGELOG.md after merging upstream NVIDIA#99, NVIDIA#107, and NVIDIA#112.

chore: merge main into fix/no-llm-negative-case

3540304

rng1995 reviewed Sep 12, 2026

View reviewed changes

Comment thread src/skillevaluator/tier3/generate_dataset.py Outdated

rng1995 requested changes Sep 12, 2026

View reviewed changes

fix(tier3): omit canned negatives for audio-domain skills

dd46ea6

Merge main and treat media/transcode skills like planning: skip guessed negatives unless eval guidance supplies one. Simplify domain overlap check for Ruff SIM103. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

mimran-khan requested a review from rng1995 September 12, 2026 14:42

chore: merge main into fix/no-llm-negative-case

aba8be2

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

mimran-khan added 2 commits September 13, 2026 10:25

fix(tier3): use only author-provided no-llm negatives

c4a49c5

Drop canned negative prompts and vocabulary heuristics that could not reliably establish an off-skill relationship. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

chore: merge main into fix/no-llm-negative-case

cd5749b

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

rng1995 reviewed Sep 14, 2026

View reviewed changes

fix(tier3): align --full contract with template case count

1c51971

Document and test that no-llm full mode emits three cases without an authored negative and four when eval guidance supplies one. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

rng1995 force-pushed the fix/no-llm-negative-case branch from e61850b to 1c51971 Compare September 14, 2026 16:22

mimran-khan and others added 2 commits September 15, 2026 10:17

docs: align README --full wording with up-to-four case contract

bd00286

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

Merge branch 'main' into fix/no-llm-negative-case

7e1d319

rng1995 reviewed Sep 16, 2026

View reviewed changes

mimran-khan added 2 commits September 17, 2026 04:20

docs(tier3): clarify --full contract for LLM vs template mode

99ebb79

Distinguish four-bucket LLM generation from template --no-llm behavior, document EVAL.md negative headings, and scope CLI mode text to no-llm. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

chore: merge main into fix/no-llm-negative-case

f8460c0

Resolve CHANGELOG conflict after catalog fleet and parallel workers landed. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

Uh oh!

Conversation

mimran-khan commented Aug 26, 2026

Summary

Verification

Release Impact

Uh oh!

rng1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mimran-khan commented Aug 30, 2026

Uh oh!

mimran-khan commented Sep 1, 2026

Uh oh!

Uh oh!

rng1995 left a comment

Choose a reason for hiding this comment

Uh oh!

mimran-khan commented Sep 12, 2026

Uh oh!

rng1995 commented Sep 13, 2026

Uh oh!

rng1995 commented Sep 13, 2026

Uh oh!

mimran-khan commented Sep 14, 2026

Uh oh!

mimran-khan commented Sep 14, 2026

Uh oh!

rng1995 Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimran-khan Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

rng1995 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

mimran-khan Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

rng1995 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

mimran-khan Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

rng1995 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

mimran-khan Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rng1995 Sep 14, 2026 •

edited

Loading