fix(chat): stop refusing queries that ask to translate - #595
Open
abdulrafey1 wants to merge 1 commit into
Open
Conversation
"translate" sat in the out-of-scope keyword list, so any query using the word without also hitting an in-scope keyword was hard-refused by the pre-filter and never reached the LLM scope classifier. The keyword layer cannot express the system prompt's "unrelated to course content" qualifier, so it over-refused: "translate this into Spanish" was rejected outright. The classifier still catches the genuine case, and "poem", "story", "fiction" and "rewrite" remain in the list. Adds the three previously-refused phrasings as regression cases, plus the research table's contrast case that already passed because an in-scope keyword short-circuits the check, and two non-English queries pinning the fall-through the pre-filter has always had but nothing asserted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
abdulrafey1
changed the title
hamza-56
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of: Sparkth UI, emails, API errors, and AI-generated content are English-only
Bottom of the phase-2 generation-language stack. Ships on its own and does not close the issue.
What
Stops the chat scope pre-filter hard-refusing any query containing the word "translate", so a
translation request reaches the LLM scope classifier that can actually judge it.
Changes
"translate"from the chat plugin's out-of-scope keyword listnon-English queries pinning the pre-filter's fall-through
How to Test
uv run pytest sparkth/plugins/chat/tests/test_scope_validation.py -v— 17 tests pass."translate"toout_of_scopeinsparkth/plugins/chat/assets/scope_keywords.jsonand re-run —test_translation_requests_reach_the_classifierfails. Revert.
met with the fixed refusal sentence.
Notes
No migration, no env var, no dependency. Behaviour change only in the cheap keyword pre-filter —
the LLM classifier still runs on everything that passes it, and
"poem","story","fiction"and
"rewrite"remain out of scope, so a genuine "translate this poem" is still refused.The keyword layer cannot express the system prompt's "unrelated to course content" qualifier,
which is why it over-refused. One of the four new phrasings —
translate this outline into Spanish— already passed before this change, because the in-scopeoutlineshort-circuits the check; it is kept as an explicit contrast case and labelled as one.This description was written with the assistance of an LLM (Claude).