Skip to content

fix(utils): reject non-finite numbers in LLM JSON loaders - #243

Merged
helloml0326 merged 1 commit into
agentscope-ai:mainfrom
kittimzhe:fix/utils-loaders-reject-non-finite
Nov 17, 2026
Merged

helloml0326 merged 1 commit into
agentscope-ai:mainfrom
kittimzhe:fix/utils-loaders-reject-non-finite

Conversation

@kittimzhe

Copy link
Copy Markdown
Contributor

Description

Third stop for the RFC 8259 rule from #226 and #232: the LLM-response JSON loaders still parse the NaN/Infinity/-Infinity literals that the JSON spec does not define.

openai_chat_model (structured-output parsing) and qwen_vl_model rely on repair_and_load_json / trim_and_load_json in openjudge/utils/utils.py. Today a complete model response such as {"score": NaN} loads as {'score': float('nan')}, and the nan silently poisons every aggregate computed from it (nan + 1.0 = nan), instead of failing into the loaders' existing error handling:

>>> trim_and_load_json('{"score": NaN}')     # on main
{'score': nan}
>>> repair_and_load_json('{"score": NaN}')   # on main
{'score': nan}

Fix: route both json.loads calls through the shared reject_non_finite helper that #232 added, and widen the except clauses from json.JSONDecodeError to ValueError (a subclass relationship — the loaders' public contract of raising ValueError is unchanged). Malformed responses now fail cleanly into the existing error paths.

Note: incomplete JSON that goes through json_repair keeps its current behavior — the library stringifies a bare NaN into "NaN" while repairing (e.g. {"score": NaN → {'score': 'NaN'}), which never produces a float; a test locks that in.

How I Tested

  • New tests/utils/test_json_loaders.py (the loaders previously had zero test coverage): 6 rejection tests across both loaders (plain and fenced NaN/Infinity/-Infinity) plus 5 behavior locks (valid JSON, code-fence stripping, incomplete-JSON repair, the json_repair stringification path)
  • Reverting the fix makes the 6 rejection tests fail (they load nan on main) — real regression tests
  • pytest tests/utils → 47 passed
  • Full suite → 1340 passed (the only failure, tests/harness/test_codex.py, also fails on clean main — environment-dependent, unrelated)
  • black / isort / flake8 clean locally

Python's json module parses the NaN, Infinity and -Infinity literals
that RFC 8259 does not define, so both repair_and_load_json and
trim_and_load_json returned {'score': nan} for malformed model output
and the nan silently poisoned every aggregate computed from it.
agentscope-ai#226/agentscope-ai#232 already enforce this rule in the JSON graders; route both
loaders through the shared reject_non_finite helper so structured-output
parsing fails cleanly into its existing error handling instead.
@helloml0326
helloml0326 merged commit 3ac7569 into agentscope-ai:main Nov 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants