Skip to content

Detect JSON input with a UTF-8 BOM - #941

Draft
Mint0424 wants to merge 2 commits into
simonw:mainfrom
Mint0424:fix/detect-bom-prefixed-json
Draft

Mint0424 wants to merge 2 commits into
simonw:mainfrom
Mint0424:fix/detect-bom-prefixed-json

Conversation

@Mint0424

@Mint0424 Mint0424 commented Nov 13, 2026 •

Copy link
Copy Markdown

Fixes #940.

Skip a leading UTF-8 BOM when checking the input sample for a JSON array or object. The original bytes still go through the JSON reader, and CSV dialect detection uses its existing input. Regression cases cover a JSON array, a JSON object after a newline, and a CSV control.

On macOS/Python 3.12, both new JSON cases reproduced the bug before the fix. After the fix, 51 targeted tests passed. A full local offline run passed 1,500 tests and skipped 16 optional tests that require the compiled SQLite example extension, SpatiaLite, or NumPy/pandas. Black, flake8, mypy, and Cog checks passed. The PR is a draft pending the upstream CI matrix.

Codex assisted with the reproduction, patch, and tests.


📚 Documentation preview 📚: https://sqlite-utils--941.org.readthedocs.build/en/941/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSON with a UTF-8 BOM is misdetected as CSV by sqlite-utils memory

1 participant