Skip to content

JSON with a UTF-8 BOM is misdetected as CSV by sqlite-utils memory #940

Open

Description

@Mint0424

On main (9c66baf, sqlite-utils 4.2.1), sqlite-utils memory misdetects a UTF-8 JSON file as CSV when it starts with a byte order mark (BOM). The JSON reader already accepts the same bytes when the format is explicit.

With Python 3.12 on macOS, run these commands in a temporary directory:

python -c 'from pathlib import Path; Path("items.json").write_bytes(b"\xef\xbb\xbf" + b"[{\"id\": 1, \"name\": \"Cleo\"}]")'
sqlite-utils memory items.json 'select id, name from items'
sqlite-utils memory items.json:json 'select id, name from items'

Actual: autodetection exits with Error: no such table: items. The explicitly typed command returns [{"id": 1, "name": "Cleo"}].

Expected: both commands return the same rows. rows_from_file() checks the first bytes for [ or { before accounting for a BOM. The fix can normalize only the detection sample and leave the actual input untouched for json.load().

I have a small patch with offline regression coverage. Codex assisted with the source review, reproduction, and patch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions