You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
JSON with a UTF-8 BOM is misdetected as CSV by sqlite-utils memory #940
On main (9c66baf, sqlite-utils 4.2.1), sqlite-utils memory misdetects a UTF-8 JSON file as CSV when it starts with a byte order mark (BOM). The JSON reader already accepts the same bytes when the format is explicit.
With Python 3.12 on macOS, run these commands in a temporary directory:
python -c 'from pathlib import Path; Path("items.json").write_bytes(b"\xef\xbb\xbf" + b"[{\"id\": 1, \"name\": \"Cleo\"}]")'
sqlite-utils memory items.json 'select id, name from items'
sqlite-utils memory items.json:json 'select id, name from items'
Actual: autodetection exits with Error: no such table: items. The explicitly typed command returns [{"id": 1, "name": "Cleo"}].
Expected: both commands return the same rows. rows_from_file() checks the first bytes for [ or { before accounting for a BOM. The fix can normalize only the detection sample and leave the actual input untouched for json.load().
I have a small patch with offline regression coverage. Codex assisted with the source review, reproduction, and patch.
Activity
Detect JSON input with a UTF-8 BOM (simonw#940)
Cover UTF-8 BOM input in format detection (simonw#940)