bug(retain): top-level JSON list fact output is rejected instead of normalized

[Lang-code-ops]

bug(retain): top-level JSON list fact output is rejected instead of normalized to {"facts": [...]}

Summary

retain/fact_extraction.py rejects semantically valid top-level JSON list outputs from retain fact extraction, even when the list contains usable fact objects.

This causes wasted retries, token spend, and queue pressure.

Expected behavior

If the parsed extraction JSON is a top-level list of fact objects, Hindsight should normalize it to:

{"facts": [...]} 

and continue through the normal validation/storage path.

Actual behavior

The list form is treated as malformed / non-dict JSON and retried instead of being accepted.

Live evidence

Recovered in production and documented in:

• 2026-06-25-hindsight-llm-config-audit/artifacts/upstream-issue-draft-retain-top-level-list-json.md
• 2026-06-24-hindsight-retain-failed-batch-recovery/README.md

Observed notes include:

LLM returned non-dict JSON ...: list. Retrying...

and later a second path-specific confirmation:

single-item extraction path still checked not isinstance(extraction_response_json, dict)
before normalizing a top-level list to {"facts": [...]}

This showed the normalization existed in one place but was ordered too late for another active code path.

Local recovery shape

The minimal fix that unblocked live recovery was to move list normalization ahead of the non-dict malformed-response branch, so a valid list is wrapped before schema rejection.

Version

Observed on live ghcr.io/vectorize-io/hindsight:latest-slim / 0.8.3-slim runtime.

Related note

This is distinct from crashes caused by malformed non-dict retries eventually surfacing None / generic exceptions. Here the payload itself was semantically usable and should not have been retried.

[nicoloboschi]

the fact llm can't follow the schema is a signal you're using a not-good-enough model so it's better to change that instead of applying workaround. even if compatible with the list, the output is probably not gonna fill all fields as expected, leading to a quality degradation of the stored memories