Is your feature request related to a problem? Please describe:
NetCDF files tend to be large. Data access is currently only marginally aware of and responsive to data size and data access patterns. Chunked data access, as available in netCDF4 files, is opaque. This leads to inefficient data access and potential memory exhaustion leading to a hard stop.
Describe the solution you'd like
Data access, both for reading and writing, should be aware of data size and possible chunking on file. Memory exhaustion must be avoided, chunking must be supported.
Relationship to the CF Metadata Conventions or other conventions
This is related to the netCDF-4 data format and general data management principles, not the CF conventions.
Are you willing and able to implement the solution
Working on it.
Is your feature request related to a problem? Please describe:
NetCDF files tend to be large. Data access is currently only marginally aware of and responsive to data size and data access patterns. Chunked data access, as available in netCDF4 files, is opaque. This leads to inefficient data access and potential memory exhaustion leading to a hard stop.
Describe the solution you'd like
Data access, both for reading and writing, should be aware of data size and possible chunking on file. Memory exhaustion must be avoided, chunking must be supported.
Relationship to the CF Metadata Conventions or other conventions
This is related to the netCDF-4 data format and general data management principles, not the CF conventions.
Are you willing and able to implement the solution
Working on it.