open_nexradlevel2_datatree fails on S3/fsspec cached Level II files (record_number is None)

[syedhamidali]

Reading NEXRAD Level II data fails when using S3 or fsspec cached files.

Example (fails):

import fsspec
import xradar as xd
file = (
    "unidata-nexrad-level2/2013/05/20/"
    "KTLX/KTLX20130520_201643_V06.gz"
)
local_file = fsspec.open_local(
    f"simplecache::s3://{file}",
    s3={"anon": True},
    filecache={"cache_storage": "./cache"},
)
dtree = xd.io.open_nexradlevel2_datatree(local_file)

Also fails with direct S3 access:

file = (
    "noaa-nexrad-level2/2013/05/20/"
    "KTLX/KTLX20130520_201643_V06"
)
dtree = xd.io.open_nexradlevel2_datatree(
    f"s3://{file}",
    storage_options={"anon": True},
)

Error

TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

Occurs in init_next_record because record_number is None.

⸻

Environment

• xradar 0.2.2
• Python 3.12
• fsspec + s3fs latest

⸻

Notes

• File downloads correctly, issue happens during parsing
• Same file works when converted to CfRadial and opened with open_cfradial1_datatree

⸻

[aladinor]

Thanks @syedhamidali for reporting this. I will try to check what is happening. In the meantime, have you tried data streaming?

Example for .gz files

import xradar as xd
import fsspec
filepath = 's3://noaa-nexrad-level2/2011/05/20/KVNX/KVNX20110520_000442_V06.gz'
storage_options = {"anon": True}
stream = fsspec.open(filepath, mode="rb", compression= "gzip", **storage_options).open()
dtree = xd.io.open_nexradlevel2_datatree(stream.read())

example for non-gzipped files

import xradar as xd
import fsspec
filepath = "s3://noaa-nexrad-level2/2023/06/29/KILX/KILX20230629_124851_V06"
storage_options = {"anon": True}
stream = fsspec.open(filepath, mode="rb", **storage_options).open()
dtree = xd.io.open_nexradlevel2_datatree(stream.read())

[aladinor]

Reproduced this locally with the exact file from the example (KTLX20130520_201643_V06.gz, downloaded from unidata-nexrad-level2).

Root cause: the file is a real gzip wrapper (1f 8b magic) around the inner AR2V0006 NEXRAD Level II archive. xradar's NEXRAD Level II backend opens local paths with plain open() + np.memmap (no gzip layer), so the memmap exposes gzip-compressed bytes, the BZ2-magic scan finds nothing, init_record(0) returns False, record_number stays None, and the next init_next_record() does None + 1 → the TypeError you see.

Confirming the diagnosis: after gunzip KTLX20130520_201643_V06.gz, the same call opens cleanly into a 17-sweep DataTree with all moments, no warnings.

The noaa-nexrad-level2 bucket serves files without the .gz wrapper, so that path works once it actually reaches a local file (note: the storage_options={"anon": True} in the second snippet is currently ignored — the backend does not route through fsspec, so s3://… is passed straight to open() and fails with FileNotFoundError before the parser is reached).

Two issues, really:

1. xradar doesn't transparently handle .gz-wrapped NEXRAD files. The recommendation is to gunzip first (the file is already on disk after fsspec.open_local's simplecache, so gzip.open / gunzip is a one-liner on the caller side).
2. The current failure mode is opaque. A TypeError: NoneType + int gives no hint that the input isn't a NEXRAD archive. Worth surfacing a clearer error from init_record / init_next_record when the BZ2 scan finds no records and/or the volume header isn't AR2V — something like ValueError("Not a NEXRAD Level II archive: no AR2V header or BZ2 records found. If this is a .gz file, decompress it first.").