Skip to content

Keep the fields of structured dtypes in the JSON codec - #871

Open
raashish1601 wants to merge 1 commit into
zarr-developers:mainfrom
raashish1601:fix/518-json-structured-dtype
Open

raashish1601 wants to merge 1 commit into
zarr-developers:mainfrom
raashish1601:fix/518-json-structured-dtype

Conversation

@raashish1601

Copy link
Copy Markdown

Fixes #518.

As pointed out in the issue, JSON.encode stores the dtype as buf.dtype.str, which is only |V<n> for a structured dtype, so the fields are lost and decode fails with ValueError: setting an array element with a sequence.

For structured dtypes the codec now stores dtype.descr instead. On decode, the descr is turned back into a dtype (JSON has turned its tuples into lists) and the records are turned back into tuples, recursing into nested structured fields. Two more details were needed for that to round-trip:

  • tolist() on a structured array leaves subarray fields as arrays, which json cannot encode, so those are converted to lists first;
  • the records are turned into tuples at the depth given by the array shape, so 0-d and n-d structured arrays work.

Non-structured arrays are encoded exactly as before, and data written before this change still decodes the same way.

TODO:

  • Unit tests and/or doctests in docstrings: test_structured_dtype in tests/test_json.py with the array from Encoding and Decoding Compound Arrays Does not work with JSON Encoder/Decoder #518 (object fields), a 0-d array, a 2-d array with a subarray field, a nested structured field and a subarray of a structured dtype. All five fail before the change.
  • Tests pass locally: tests/test_json.py, tests/test_zarr3.py and the json doctest, built in a Linux container.
  • Docstrings and API docs for any new/modified user-facing classes and functions (no API change)
  • Changes documented in docs/release.md
  • Docs build locally (not built; only a release note line was added)
  • GitHub Actions CI passes
  • Test coverage to 100% (Codecov passes)

@codecov

codecov Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (1f83681) to head (c5d9732).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #871   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines          905       935   +30     
=========================================
+ Hits           905       935   +30     
Files with missing lines Coverage Δ
src/numcodecs/json.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Encoding and Decoding Compound Arrays Does not work with JSON Encoder/Decoder

1 participant