Proposal: Reframe NetCDF as a reference structural encoding of CF so other structural layers can be defined.

[dblodgett-usgs]

Topic for discussion

Proposal: Reframe NetCDF as a reference structural encoding of CF so other structural layers can be defined.

This proposal follows from #463 and #484. It aims for minimal and incremental modification to enable CF to be supported by structural encodings beyond NetCDF. It prioritizes compatibility with existing NetCDF and Zarr data but also refines CF in a way that cleanly separates it from NetCDF, unblocking the potential for future structural layers.

1. Premise

CF defines constructs (what a coordinate variable, a cell method, a grid mapping means) and relies on a structural data model to hold them (dimensions, variables, attributes, groups). Today the structural model required to conform with CF is netCDF, required by a single sentence in Section 1.3: "The terms in this document that refer to components of a netCDF file are defined in the NUG."

Appendix I describes the netCDF encoding "CF-netCDF" and establishes CF independence from the NetCDF encoding as a design goal (Hassell et al. 2017).

This proposal makes the NetCDF encoding of CF explicit and substitutable. It evolves CF to state the components it requires of any structural layer, keeping the (NetCDF User's Guide) as a reference encoding, and scopes the handful of sections that describe netCDF as such.

This proposal does not change any CF semantics, restructure the document, deprecate the netCDF encoding, or define a Zarr (or other) encoding within CF.

2. Structural layers

• NUG (NetCDF User's Guide) — the reference layer; components, attribute conventions. Nothing changes for CF-netCDF datasets.
• NZ (zarr-conventions/nz) — a structural layer on Zarr v3 based on xarray encoding assumptions. NZ is written to supply the same components the NUG supplies (named shared dimensions, typed arrays incl. scalars, group/array attributes, _FillValue, dimension coordinate arrays). It documents what xarray already writes and is registered under the zarr-conventions framework.
• Designed to admit others on the same terms, e.g. a tabular layer on Parquet / GeoParquet for Discrete Sampling Geometries (ch. 9; cf. @ethanrd in Separating CF semantics from the netCDF encoding #484), or OGC CoverageJSON, whose domain axes and typed ranges are the same components.
• How a structural layer is defined:
  • Its own documentation conventions state how it supports each required component.
  • A dataset declares both in its Conventions declaration (e.g. "NZ-1.0 CF-1.14").

3. The edits to be applied in one PR

Section 1.3 Terminology (+ one sentence in Section 1.1).

Replace the NUG-binding sentence with the list of components CF requires of a structural layer:
1. named, sized dimensions shared within a group
2. typed N-d variables incl. scalars with an ordered dimension list
3. attributes at variable and group/global level
4. a root group (groups optional)
5. coordinate variables (1-D, name = dimension)
6. reserved _FillValue.

Define the NUG as the reference encoding.
Define "most rapidly varying dimension" by declared (CDL) order rather than storage.

Why: This core dependency is CF's only required binding to netCDF. Spelling out the contract with the NUG makes it substitutable. Declared-order definition allows the "(CDL order)" rule in ch. 7, ch. 9 and the conformance doc to go unchanged.

Chapter 2 scoping

Chapter 2, "NetCDF Files and Components" will need to be reframed to emphasize that NetCDF is a reference implementation of the CF constructs.

1. title/preamble become "Components of a CF Dataset" framed using the NUG as the reference implementation.
2. Section 2.1 filename needs an addition that other encodings identify a dataset by their own means (a Zarr hierarchy by its root metadata, for example).
3. Section 2.2 data types incl. strings needs subtle changes indicating what CF requires and that NetCDF is a reference encoding of it that takes its own opinions.
4. Section 2.3 names requires only light framing changes to imply that NetCDF is an example encoding.

Regarding strings, CF already defines the base requirement and implementation options as a "may be" statement: Section 2.2 says an n-dimensional array of strings "may be implemented as" a string variable with n dimensions or a char variable with n+1. Both netCDF forms (NC_STRING, and NC_CHAR with a trailing string-length dimension) and both registered Zarr v3 string types (string, fixed_length_utf32) are arrays of Unicode strings, and they map losslessly in both directions. The edit therefore would state the base requirement and that in netCDF, it is stored in the two existing forms.

Why: These sections describe the netCDF structural encoding of CF as an exemplar. We have verified that xarray's Zarr v3 output for CF-netCDF string variables already lands on registered Zarr types, so no new string concept or deliberation is needed. All other adjustments are reframing to provide clarity that NetCDF is the reference encoding.

Section 2.5.1 Missing data

Restate the missing-data rules in CF rather than deferring to NUG Appendix A:

1. Values equal to _FillValue (a scalar of the same type as the stored data) are missing.
2. Values outside valid_range, or below valid_min or above valid_max, are missing.
3. Values equal to missing_value are missing.
4. When a single missing value suffices, _FillValue is the recommended attribute.
5. Where the structural encoding defines a default fill value for a data type — as the NetCDF library does — values equal to that default are treated as missing when _FillValue is absent.

The CF rules already in this section (missing data permitted in data and auxiliary coordinate variables but not in coordinate variables; missing values interpreted before unpacking; actual_range) are unchanged. The paragraph on the NUG 2.3 to 2.4 change and the description of the NetCDF library's pre-fill behaviour remain as NetCDF-scoped notes.

Why: This section is the heaviest deference to the NUG in the document — CF's definition of "missing" currently lives in a document CF does not version or control, a point @larsbarring raised in #463. @JonathanGregory proposed stating these rules in CF in his #463 summary. Item 5 preserves current CF-netCDF reader behavior while making explicit a property another structural encoding may lack: Zarr's storage fill_value initializes unwritten chunks and carries no masking semantics, so an array without _FillValue has no missing-data indicator there.

Section 2.6 Attributes (incl. Section 2.6.1)

1. Section 2.6.1: Conventions is the root-group attribute by which a dataset declares the conventions it follows. CF already states the list syntax (blank-separated, or comma-separated when a name contains blanks), so the sentence attributing that syntax to NUG Appendix A becomes an informational note.
2. Section 2.6: add one paragraph for structural encodings that do not declare attribute data types. CF's typing rules are written against declared attribute types — _FillValue, missing_value, valid_*, flag_values and flag_masks take the type of their variable; actual_range takes the unpacked type; scale_factor and add_offset must be float or double and determine the unpacked type. Proposal: Where attribute types are not declared, the same-type rules resolve from the variable's data type, and scale_factor and add_offset are taken as double.

Why: Without this paragraph, CF's typed-attribute rules cannot be applied where attributes are JSON values, which is the case for Zarr. The double default is the guidance Section 8.1 already gives for packed data whose attribute types do not conform ("unpacked to double type, in order to minimize the risk of loss of precision"), so existing CF-netCDF datasets don't read differently.

Section 2.7 Groups

1. Restate the section on structural layers, such as NetCDF4, that provide group functionality: a root group, a tree of named child groups, group-level attributes, and variables and dimensions that belong to one group and are identified within it by name. A layer without groups would be treated as a "root-only" case.
2. Replace "the same netCDF dimension ID" with a definition of dimension identity that does not depend on an ID. Name and length can be used instead of ID.
3. The quotation from the NetCDF groups documentation becomes an informational note on how NetCDF scopes dimensions rather than the principle the rules are based on.

The search strategies (absolute path, relative path, proximity, lateral) and the rules for Conventions, external_variables, title and history in non-root groups are defined within CF and do not need to change.

Why: Section 2.7 came in with CF-1.8 and is CF's own name-resolution convention; the dimension-ID clause is its only NetCDF-bound sentence. Stating identity by name and length is what a layer that scopes dimension names per group needs in order to apply the section at all.

Reference wording

1. Section 3.1 already limits CF's dependence on UDUNITS to "the definition of legal units strings" and states that CF does not require the UDUNITS software for conversion. Align Section 1.4 and the bibliography entry with that scoping — the reference is to the UDUNITS unit grammar and database, not to the software package.
2. Section 8.1: strike "At the current time the netCDF interface does not provide for packing data," which describes a library rather than the convention.
3. Conformance document: mirror the Section 2.7 change in its own 2.7 section (the dimension-ID sentence, and "NUG-coordinate variables" becomes "coordinate variables").

Why: These are alignments with scoping CF already states, not changes in meaning.

Not changed

Appendix A's cross-references to NUG Appendix A stay as attribution — CF defines every attribute it requires in its own text (Sections 2.5.1, 2.6.2, 3.1, 3.2 and 8.1), so the citations record provenance rather than deference. Appendix I is unchanged; it already describes the encoding-independent data model this proposal relies on. CDL remains the notation for examples. The unlimited-dimension mentions in Chapter 9 and Appendix H are already conditional on wanting to append and do not need to be edited; the same holds for Section 8.4, which presents the NetCDF library's quantization attributes as one implementation among others. Incidental "netCDF file" wording in Chapters 4, 5, 7 and 9 and Appendix D can become "dataset" as an optional follow-up.

4. Why this path

Parsimonious. Six sections and one PR that do not introduce ideas beyond what Appendix I already introduces, "CF-netCDF", and every CF-netCDF dataset remains valid verbatim — the condition Section 1.2 item 10 sets for any change.

Incremental. Each edit can be checked as a thought experiment against data and software that exist today — netcdf-c, xarray, cf-python, the cf-checker — rather than as a restructure every reader and every training document has to absorb before its value can be judged.

Compatibility-first. The structural layers are written from what implementations already do: the NUG from NetCDF, NZ which documents xarray's Zarr V3. Conformance of existing data against a structural layer defined independent of CF is therefore testable now rather than after a new encoding is designed and adopted.

Progressive. The edits unblock the use cases raised in #484 — ESA's Sentinel archive on Zarr, Copernicus ARCO ERA5, ERDDAP's Parquet output — without limiting the potential to pursue a more significant Part I / Part II restructure, a formal grammar, or other formalization later. Each of those will be easier once the binding between CF and its structural layer is explicit.

Governance-light. CF defines the components it requires. Who publishes an encoding for a given container, and whether that encoding is in CF as an appendix, remains a decision the community can make case by case, as discussed in #463.

5. Proposed next step

I will draft the PR against cf-conventions and open it for discussion, with the Section 1.3 component list and the Section 2.5.1 and 2.6 restatements as the parts most in need of review. I would ask the maintainers of the groups material (Section 2.7), the Discrete Sampling Geometries chapter, and the conformance document to review the respective edits. The target is CF-1.14.

Three questions for this thread:

1. Does the six-component list in Section 1.3 capture everything CF depends on from the NUG? If a chapter relies on a NUG property not in the list, that is a seventh component rather than an argument against the approach.
2. Does the attribute-typing paragraph belong in Section 2.6, with the other attribute rules, or in Section 8.1 with the double guidance?
3. After this PR, should the NetCDF-scoped sentences be collected into an appendix — a small step toward the reference-encoding part of the Separating CF semantics from the netCDF encoding #484 proposal — or stay in place?

LLM Agent Disclosure: Claude Fable 5.1 was used to assist research and outlining this discussion entry. The outline was filled in and edited by @dblodgett-usgs. All LLM generated text has been reviewed and validated.

"Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government."

[JonathanGregory]

Dear Dave

Thanks for making this proposal. I take this as a pre-proposal, or an outline proposal, since it's not a conventions issue. Is that the idea? As I say in commenting in Patrick's discussion, I prefer an incremental approach, such as this one.

I argue also that any change should be designed as sufficient for the current need and that, for the moment, our need is to modify the CF conventions document to make it neutral between netCDF and Zarr, since those are the only formats for which it's actually being used. I understand your stated aim to separate CF cleanly from netCDF, but I think it will help us to keep in mind that Zarr is the present need. That will clarify what "separating cleanly from netCDF means" for this purpose. It sounds like some other formats, which we have not yet begun to consider in detail, might differ more fundamentally, and thus require a more radical "separation". For example, tabular formats might not have a way to support hierarchical groups. But let's not think about such difficulties now (unless someone advocates that we should). I would further argue that we should specifically include in ouour aim to say how CF maps to Zarr. Unless that's part of it, I'm not convinced we should make any change at all. The document is already fine for netCDF alone!

The aim I'm suggesting is thus a bit broader than yours, but not huge! I think it would be helpful to develop it in a branch, rather than a series of independent PRs, because I suspect that there will be interactions between different aspects of it. Also, doing it as a branch in the CF repo would enable collaboration. I agree with your warning about forking CF. I definitely don't want that to happen, so I think it's important to be clear about the timescale. You suggest CF 1.14. That means we have to agree it by a week today, which I feel is too soon. I would suggest CF 1.15, giving us a year, but there's no reason to delay. The branch should be merged back into main as soon as it's ready.

In Geozarr issue 138 I suggested various things we could do towards netCDF/Zarr-neutrality (arising from a discussion with David). Here's that list, with a few additions and changes in the light of your proposal. Some of them are included in your detailed proposal.

• Remove "NetCDF" from the title of the conventions document. Note that the CF website is already entitled "CF metadata conventions", not mentioning netCDF. Remove "netCDF" from the text elsewhere.

• Replace the word "file" with "dataset". We should do that anyway, regardless of Zarr and considering only the netCDF API, because the data is not necessarily in netCDF files.

• Insert statements in the Abstract and at the start of the Introduction that CF is a metadata convention which can be used for both netCDF and Zarr datasets.

• Confine the few netCDF- and Zarr-specific points to special sections of the document. As you suggest, section 2 is probably where this could be done.

• Remove references to the NetCDF User Guide, which is cited only as an authority (rather than necessary to be read) in nearly all instances; the requirements of the NUG as applied to CF are nearly all explicit in the CF conventions document already, and it would be better to do this thoroughly (as you say about missing data). We also need to describe the data types required for CF (as you say).

• Remove most of references to COARDS, if I'm right that this is no longer a concern for many datasets - am I?

• Rearrange Appendix I to describe the storage-format-neutral CF logical data model before describing the encoding in netCDF or Zarr.

• Clarify what constitutes the CF semantic data model, at the points where its components are described. I think this includes the separation of metadata into standard name, cell methods and coordinates (which we do not currently explain at all), and the various controlled vocabularies, such as the attribute names of appendix A.

• Make the terminology work just as well for Zarr as for netCDF. In Bonn, I suggested using the terminology of the CF data model e.g. "dimension coordinate construct" rather than "coordinate variable [in the NUG sense]", "field construct" rather than "data variable". However, reservations were expressed about "construct", because it's unfamiliar and could be off-putting. But "variable" is a netCDF-specific term, and Zarr users would prefer "array", I suppose. If we can't find an obvious common terminology, maybe we could get round this by providing the document in separate netCDF and Zarr dialects, from a common Asciidoc source.

• The last would also be a way to provide examples in both CDL and Zarr-json. You could argue that CDL examples are fine for Zarr users, if we assume they're always going through the netCDF API, but I'm not convinced about that. What do you think?

Best wishes

Jonathan

[dblodgett-usgs]

Dear Jonathan,

We are in broad agreement here. I consider what I laid out to be a proposed direction with a fairly specific strategy. I'm find working it in a branch with a clear expectation of it not continueing for very long. The version 1.14 was an oversite on my part -- that should read 1.15.

As time allows, I'll draft up an initial cut at this that takes your suggestions into account and push it up in a fork that we can migrate into a branch of the convention repository and we'll see where things go from there.

Regards,

Dave

[ChrisBarker-NOAA]

Just a note on gitHub usage:

A branch in the main repo is absolutely the way to manage significant changes like this -- I suggest getting this work into a local branch sooner than later -- there is no reason to wait.

[Git is great for this sort of thing, whatever happens in a branch has no effect on anything else....]

A key limit, however, is how many people have push permissions on the main repo ... anyone that is making significant contributions should, or is become a mess ...

[JonathanGregory]

Yes, I agree that updating a branch only by PRs from private repos forked from the branch would be a clear and well-controlled process. As far as I can see, you can't set different permissions on a specific branch, which we'd need to allow PRs to be merged by people who can't push to main in the conventions repo. Could we instead fork a new repo in the GitHub organisation? I guess then it could have different permissions from the conventions repo.

[ethanrd]

I agree with Dave, for ease/consistency of change tracking, all changes should be managed through PRs.

The CF conventions repo main branch is marked as protected but I'm not sure if PRs are required. If not, I think we should add that. I would push for us to do the same with all development branches like we are discussing for this work.

[ethanrd]

Besides convenience and speed, are there other reasons more people need to be able to merge PRs?

And a more general comment. Please! Let's not do any ad hoc changes to how CF manages GitHub permissions. The CF GitHub permissions are currently based on a person's role(s) in the CF Governance structure (committee, panel, team, etc). Assigning permissions to individuals will make the whole permissions scheme much harder to manage.

[ChrisBarker-NOAA]

The CF conventions repo main branch is marked as protected but I'm not sure if PRs are required.

I'm pretty sure that's what protected means -- you can't push directly to it (at least most people can't).

all changes should be managed through PRs.

You can (and should) do PRs within the same project / repo -- that is what I'm advocating for -- they do not have to be from a separate Fork.

It's hard to explain, but the overhead really is MUCH larger if you have a team that's working in a bunch of separate forks, rather than using branches within one project.

I'm not suggesting anything radical here -- it's standard practice that a project has a core team of folks that all work within the repo, and Forks are used for "outsiders" making contributions -- that's all I'm suggesting.

Let's not do any ad hoc changes to how CF manages GitHub permissions.

I agree with that -- I only think we should use the the main repo if the current permissions structure will support the project.

NOTE: if folks really think that a pile 'o forks will work well, we can give it a shot, if it turns out I'm right, we can always move to a different process later -- that's part of the glory of git!

[ethanrd]

Hi Chris

I'm pretty sure that's what protected means -- you can't push directly to it (at least most people can't).

That's what I thought as well (and maybe that was the case when the protected branch feature came out). Looking at the GitHub docs, requiring PRs is one of a variety of options and none of them are required.

it's standard practice that a project has a core team of folks that all work within the repo, and Forks are used for "outsiders" making contributions -- that's all I'm suggesting.

Agree. I'm just still not sure about the "pain". In my experience, creating a PR from a branch I push to my fork (of whatever project I'm working on) is no harder than creating a PR from a branch I push to the projects origin repo. And it works whether I have write permission on that project or not. Reviews and suggesting changes works the same so I generally don't even notice if a PR is based on branch on the project repo or on an individual's fork.

Anyway, I think we've gotten pretty deep in the GH workflow weeds here. I will just reiterate that changing how CF manages GitHub permissions, including creating a new CF repo with different permissions, adds complexity. I think we should avoid that if at all possible, especially permissions based on individuals or particular projects/efforts.

[ChrisBarker-NOAA]

updating a branch only by PRs from private repos forked from the branch would be a clear and well-controlled process.

But a serious pain in the @#$ -- PRs from forks work great for small self-contained changes from one person, they are a nightmare for collaboration on large refactor.

Note that git flow (or other workflows) requires branching and controlled merging, it does not require the branches be forks...

As far as I can see, you can't set different permissions on a specific branch, which we'd need to allow PRs to be merged by people who can't push to main in the conventions repo.

But you can protect a branch, so folks can't push to main.

I personally believe in-a-trust-folks philosophy -- is there anyone we that we want to make significant contributions to this effort that we don't trust to no mess with other things?

If so, then:

fork a new repo in the GitHub organisation?

Is a good solution -- we can give that special temp repo a broader set of team members.

[ethanrd]

Hi Chris,

But a serious pain in the @#$ -- PRs from forks work great for small self-contained changes from one person, they are a nightmare for collaboration on large refactor.

I'm not sure what the pain is you mention. This is what we do currently on the main branch and it seems to work fine. There would be no difference to doing this on a development branch like we are discussing here. Or am I missing something?

[pvanlaake]

Dear Jonathan, Dave and Chris,

Thanks, Jonathan, for setting out your list. A good part of it is also what the #484 proposal aims for:

• removing netCDF from the title and "file" from the text;
• making the requirements that are now delegated to the NUG explicit in CF;
• rearranging Appendix I so that the storage-neutral logical data model comes first;
• clarifying what constitutes the CF semantic data model;
• terminology that works beyond netCDF;
• examples in more than one encoding, generated from a common source.

Where we still differ is on the architecture: incremental adjustment of the current document, or a document built from the outset on a separation between semantics (Part I) and encodings (Part II, with netCDF as the reference encoding). So far that difference has mostly been argued as a matter of preference. I don't think more argument will settle it. Concrete examples will. I therefore propose that we work in two branches/repos, both taken from main:

1. The branch/repo Dave proposes, with targeted, incremental edits along the lines of this discussion and Jonathan's list.
2. A second branch/repo, which I will maintain and open to anyone else who wants to contribute, with a two-part structure: Part I describing the CF semantics independently of any container, Part II describing the netCDF reference encoding, and, as a test of the separation, an encoding of the same semantics in a second container (Zarr, using GeoZarr conventions). This branch/repo will also try out a different presentation: a hyperlinked site with a "Getting Started" entry point for newcomers, from which the Parts can still be rendered as PDF. This is to acknowledge the substantial discussion on finding more appropriate ways to present the CF conventions to users during the breakout session at the recent CF Workshop.

To be clear about the second branch/repo: it is a study for comparison, not a fork of CF and not a competing submission for CF-1.15. Its purpose is to let the community judge both approaches on actual text and format. Whatever is found useful can be moved into main through the normal process.

For the comparison, I suggest two criteria:

• Conformance preservation: every dataset that conforms to the current CF-netCDF conventions also conforms under the new text, with the same meaning.
• Container independence: the semantics can be expressed in a container other than netCDF without that container having to emulate the netCDF data model.

Jonathan, I agree that enabling CF on Zarr is the present need. But the second criterion is exactly what will tell us whether a change made for Zarr is a real separation or a restatement of netCDF in other terms. Ethan's example of Parquet for DSG data in #484 shows that the question is not hypothetical.

On the mechanics, I'm happy to follow whatever is agreed here for Dave's branch/repo, whether that is a protected branch in the conventions repository or a separate repository in the organisation. Both branches/repos should be governed the same way. My aim is to have a substantial part of the second branch/repo ready for discussion well before the 2027 Workshop, so that the comparison can be part of the programme there.

Best regards,
Patrick

[ChrisBarker-NOAA]

Thanks @pvanlaake -- this is a helpful boiling down of the process a couple tiny notes:

examples in more than one encoding, generated from a common source.

hmm -- it will be interesting to see how that plays out in reality -- it could just make things more confusing.

I actually think that CDL is a pretty good "psuedo-code" that captures the key points in a clear an understandable way, even for folks that are not NetCDF-fluent.

And I can't think of any other encoding that would be as clear in human readable text form (maybe JSON?) -- however, as I write this, I'm reminded of a parallel example:

In the Python packaging docs, they describe how to do the same thing with multiple tools, and have swtichable-tab system to easily let the user choose which they want to look at, e.g. :

https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#declaring-the-build-backend

So that could work nicely. I think worth exploring, in any case.

As for the two-pronged approach -- it seem like a lot of duplicative work to me -- but if you want to so it, I agree that a fleshed out example is probably the best way to make a decision.

Though:

as a test of the separation, an encoding of the same semantics in a second container (Zarr, using GeoZarr conventions).

Frankly, The Zarr data model is so close to netCDF that I don't think this would really demonstrate much -- is there another more different encoding that we could use as a test-case?

[pvanlaake]

Hi Chris, thanks for your comments. I had the switchable tab model in mind for this, but more likely it will be multiple links to container-specific Part II sub-sections. I do not think that examples of one concept in multiple encodings is going to be problematic. The typical user will be interested in just one encoding and go for those examples. If we get GH issues asking about why this CDL example does not align with that JSON snippet then we can politely point out that apples are not to be compared to pears or oranges.

Test of separation: @ethanrd has suggested Parquet.

[ChrisBarker-NOAA]

I'm not sure what the pain is you mention. This is what we do currently on the main branch and it seems to work fine.

As I said, it works fine for PRs for a small self contained change from one person.

But if you are doing a major re-working with a group, each separate person has their own fork, and that only they can write too (without another permission-heck process) -- and then you have many different version that constantly have to be rebased. It is substantially more overhead than letting multiple folks push directly to the working copy.

Or you can have "feature branches" for a particular change, and more than one person can work on that branch as well.

[ethanrd]

Sorry Chris. I'm still confused. Doesn't every push by anyone else cause the need to rebase or merge, with potential conflicts?

Also, doesn't the ability to review and suggest changes to PRs alleviate a lot of these issues?

[dblodgett-usgs]

Hi All -- I've created a branch https://github.com/cf-convention/cf-conventions/tree/no-netcdf -- I will moderate PRs to that branch in the interest of this goal.

My plan is to draft initial content based on my detailed proposal and the follow up discussion. I will open a PR against main as a work in progress where we can review and comment on specific details of the plan.

[JonathanGregory]

Dear Dave, Patrick et al.

Thanks for the branch, Dave. I've reopened this discussion because I think some useful conversations have been taking place in it, which might continue fruitfully, and anyway it's useful for reference. I hope that's OK with everyone.

In addition, I expect we will need a place to discuss what we are doing in this project, as it proceeds. Discussion in PRs is helpful and convenient when it's about the wording in individual places in the text, but if conversations begin about more general issues it can get very confusing, with several threads going on in different places in the PR. I think we ran into that difficulty with the very complicated PRs for leap seconds, for example. For those reasons we advise against substantial discussions about content in PRs, in CONTRIBUTING. I fear that saying this sounds pernickety, but it's not for the sake of rules: I really think it helps to keep discussions in issues.

I will think about several PRs for the detailed points I listed above.

Ethan asked

Besides convenience and speed, are there other reasons more people need to be able to merge PRs?

It feels more collaborative, I would say, if any participant can merge their own work - once everyone has agreed, of course. But I agree we don't want to complicate the permissions in the organisation, which you've worked hard to sort out. Let's see how we get on.

Patrick wrote about "try[ing] different presentation: a hyperlinked site with a Getting Started entry point for newcomers, from which the Parts can still be rendered as PDF. This is to acknowledge the substantial discussion on finding more appropriate ways to present the CF conventions to users during the breakout session at the recent CF Workshop." Thanks. That was a useful discussion, and it will be interesting to see your ideas to that end.

Patrick also commented,

Jonathan, I agree that enabling CF on Zarr is the present need. But the second criterion is exactly what will tell us whether a change made for Zarr is a real separation or a restatement of netCDF in other terms. Ethan's example of Parquet for DSG data in #484 shows that the question is not hypothetical.

I understand the point, and I agree with you and Ethan that there are other possibilities. My suspicion is that there isn't a unique, absolute, well-defined way to state CF independent of encoding. The degree of generality needed depends on what we're trying to encompass. My point is that we shouldn't go further from our present netCDF-API-like ways than we need to, because everyone is used to things as they are. If there's no gain (in terms of being able to do something we need to do), why should we cause pain to users of the CF document?

Chris wrote,

In the Python packaging docs, they describe how to do the same thing with multiple tools, and have swtichable-tab system to easily let the user choose which they want to look at

Yes, that sort of thing is what I had in mind for CDL and Zarr-JSON. But there's no need for it if Zarr users of CF are perfectly happy with CDL alone.

Best wishes

Jonathan

[dblodgett-usgs]

Dear Jonathan,

I'm fine reopening the discussion but would actually prefer to move discussion of finer points to issues. My thought was to form a draft in a branch and use a PR as a way to view the diff then draft issues if follow up discussion is required.

I do not expect the edits to be significant enough to warrant a word-processor track changes but if we look at the PR in total could port over to a google doc.

All the new open discussions here have wandered from the point of the discussion so I felt that closing it in favor of a more focused discussion would be wise. I'll refrain from opening further threads here but am happy to follow up if others want to continue duscusson here.

Dave

[ethanrd]

I agree, the different threads in this discussion are getting a bit interwoven (sorry! I've certainly contributed to that) and hard to track.

[ChrisBarker-NOAA]

Discussion in PRs is helpful and convenient when it's about the wording in individual places in the text, but if conversations begin about more general issues it can get very confusing, with several threads going on in different places in the PR

Good point -- this may be a good argument for making a fork of the repo in which to do this major update / re-write -- treating it as its own project, and then we'll have issues, etc that can be used.

But we can wait and see how "major" it is.

@pvanlaake's version is major -- I think that one should go in a separate repo.

[pvanlaake]

Dear Jonathan,

Thanks for reopening the discussion, and for your comments.

On whether there is a well-defined way to state CF independent of encoding: I agree there is no unique, absolute one. But CF has already established its level of generality. Appendix I describes the CF data model as an interpretation of CF that is "as far as possible independent of the netCDF encoding", and the data model paper presents it as describing all CF-compliant data. Stating the conventions at the level of the data model's constructs is therefore not an arbitrary abstraction. It is the target the community has already adopted. The separation I propose is relative to that model, not absolute.

On the pain to users, we should recognise that CF is currently offered in just a single encoding flavour: netCDF. If another encoding is added then that will have no impact whatsoever on the current users of CF. New users of CF preferring Zarr for their data are subject to double pain, however: first to extract the CF semantics from its current entanglement with the netCDF encoding; and secondly, and way more pernicious if you ask me, having to navigate the choices that data producers of Zarr data repositories have made in the absence of normative guidance from the CF community. Restructuring the conventions does have a cost for readers who know the current document well. That cost has to be weighed against the difficulty newcomers have with the current text, which was a recurring theme at the breakout session.

On the gain: you write that dual examples are not needed if Zarr users are happy with CDL. Many Zarr users do not write through the netCDF API at all. As an example, GeoZarr, which ESA is adopting for its Sentinel archive, organises its metadata differently from the netCDF data model, so CDL does not describe what these users write. For them, a CF statement that is not expressed in netCDF-API terms is a real gain, not a theoretical one. It would be good to hear directly from those communities in this discussion - we of course already had a presentation from the GeoZarr chair at the workshop, as a starting point.

Finally, on the practical side: could the maintainers confirm whether a second branch, taken from main, can be set up for the two-part study I described above, under the same arrangements as Dave's? If that is not convenient, I will develop it in a separate repository and report progress here and in #484.

Best wishes,
Patrick

[pvanlaake]

@ethanrd wrote:

Yes, we can create such a branch. Do you have a name in mind?

I would go for devel-semantics.

I've looked a bit into git mechanics after @ChrisBarker-NOAA 's comment and I think that a clean repo under the CF org would be the best option. Git will have difficulties maintaining the diffs (which will be substantial) and so the whole thing reverts back to separate files - clean start sounds like the better option to me. That does mean that I will have to track provenance of the text in my individual files. A chore, but doable.

Just to clarify: I am not intending to dramatically rewrite the conventions text, just organise the material differently. Semantics and encodings go in separate files and the current chapters will be broken up into more manageable (user's viewpoint) topics.

[ethanrd]

Hi Patrick @pvanlaake - I have created a devel-semantics branch from the main branch on the conventions repo.

I am unclear what you mean by git having difficulty maintaining diffs. It can certainly be difficult looking at diffs when major changes (especially across files) have been made in a single commit. On the other hand, breaking big changes into a series of small change commits with clear commit messages can help communicate the steps/changes that have been taken and make the diffs more bite sized. It may be a bit more work on the git change management side of things but it is a useful way to capture/communicate the large scale changes being made and their intent.

If you mean how git actually tracks the different commit versions, I believe it is more complicated than either diffs or full files. GitHub/Git uses a variety of optimizations, often done in the background, for tracking/storing both big and little changes. So I don't think that is an issue we really need to worry about.

A new repo adds some GH maintenance burden and means there is no potential for any git history. A development branch is a lighter maintenance burden, allows for a git history, and the git history/diffs can be ignored if they aren't useful. So I'm not sure what advantage a new repo really provides.

Happy to keep discussing. I'm certainly not a GitHub expert so may very well be missing something. I am going to ping the CF Information Management and Support Team (@cf-convention/info-mgmt) to get more thoughts on the matter from those tasked with GH maintenance.

[dblodgett-usgs]

In the past, it may have been in the era that CF was migrating to github in the first place, we discussed some guidance for contributions that are fairly major. At the time it was best practice to use a word processor for any changes would be clumsy as a code diff. The vscode and github diff view has come a long way since then and does a much better job diffing edits in the middle of a line. But the UX of a google doc or word doc is still superior to a git-based asciidoc/markdown editing workflow in many cases.

For a large document reorganization, it may be best to port out to a word processor as a result. It really depends on who is going to be doing the work though.

[pvanlaake]

Thanks a ton @ethanrd ! I'll fork the branch and work off that.

For the PRs on the branch that will be coming: I hope that this can be an expedited process. Nothing will be pushed into main and this is only a development branch, after all. Is there a process in place for that, or will this be on an ad-hoc basis?

[ethanrd]

There was some discussion of the rules for approval below and Jonathan spelled out (last paragraph of this comment) a possible starting point that slightly loosens the existing rules.

[ethanrd]

Hi Dave @dblodgett-usgs

Just a quick comment on the missing data section of your proposal.

Restate the missing-data rules in CF rather than deferring to NUG Appendix A:

• Values equal to _FillValue (a scalar of the same type as the stored data) are missing.
• Values outside valid_range, or below valid_min or above valid_max, are missing.
• Values equal to missing_value are missing.
• When a single missing value suffices, _FillValue is the recommended attribute.
• Where the structural encoding defines a default fill value for a data type — as the NetCDF library does — values equal to that default are treated as missing when _FillValue is absent.

As a netCDF library defined attribute [1], I think _FillValue should not be part of the format agnostic part of the specification. Or at least not the recommended attribute.

[1] From the NUG: 1) "Attribute names commencing with underscore ('_') are reserved for use by the netCDF library" and 2) "The netCDF library itself does not handle [missing data] in any special way, except that the value of a _FillValue attribute, if any, is used in pre-filling unwritten data."

[dblodgett-usgs]

This section will require some care -- I know that the semantics of fill values and missing data have been a stumbling point. I'll have to refresh my memory on the details but will be sure to account for this one when I get to drafting up the first draft.

[JonathanGregory]

I'm wondering if your proposals would be easier to consider as a series of PRs to the branch, rather than all in one PR, Dave? The missing data could be one PR, for example. The various things I mentioned are mostly independent, and I feel that separate PRs would be appropriate for them.

[ethanrd]

Or an issue for each concept/section for discussion and then associated PRs? Though I suspect an initial draft by Dave followed by issues raised against that draft might be easier (both for Dave and for the rest of us).

Also, would it be useful to add labels for the two un-netCDF-ing efforts/branches so we can label issues and PRs with them? (Here's were a defined process around how to deal with development branches might be useful. ... So we should remember to write down the process we settle on here.)

[dblodgett-usgs]

Let's see how this proliferates? A label or two might be a good idea.

[JonathanGregory]

Dave @dblodgett-usgs, thanks for opening conventions issue 677. I will contribute to it later.

In the way Dave has written it, his issue (677) will become the proposal to merge his branch no-netcdf back into main. He writes that 677 is intended for process, goal and sequence. He envisages that other issues will be opened, linked to 677, to deal with particular aspects of the work in detail. I presume that all these subsidiary linked issues will be concluded and their PRs merged, to update the branch, before the final step of proposing to merge the branch into main.

I think this is a good procedure, for this new kind of situation. I suggest that we shouldn't "legislate" excessively about this in advance, because we don't know how it will work out, but that it's good to be clear at any time what our agreed procedure actually is, and revise it by consensus if necessary as we go along. If it's successful, we can write it later in CONTRIBUTING for the record, in case it's needed again.

Dave and @ethanrd have discussed the process for updating the branch. They both think that it should be similar to the process we use for usual issues to update main. Dave suggests perhaps we could "lower the stakes" a bit, because there will still be a subsequent process for acceptance. I suggest that modifications to the branch should be supported by at least two people (i.e. the proposer and someone else) rather than three, that neither has to be a committee member, and that no new concerns should be raised for one week (rather than three) before merging. What do you think?

[dblodgett-usgs]

Thank you for pulling this together, @JonathanGregory -- I've adjusted the description on cf-convention/cf-conventions#677 to reflect the specifics you described -- very much in line with what I was envisioning.

[JonathanGregory]

Patrick @pvanlaake, I agree that diffs are quite difficult to understand for large-scale rearrangements, since they're shown as deletions and additions (of equal text).

For humans to understand, it occurs to me that it would be most digestible if you could produce some diagram or table, which lists on the left the current sections and order of contents, and on the right the proposed rearrangement. That will enable us easily to see and think about the mapping.

To implement it clearly within a branch, you could perhaps:

• Split the existing ch*.adoc files into smaller ones.

• Modify cf-conventions.adoc to link them all in the present order e.g. replacing include::ch02.adoc[] with include::ch02partA.adoc[], include::ch02partB.adoc[] etc. This will regenerate the existing document with no change.

• Reorder the includes in cf-conventions.adoc to produce the new order.

• Any necessary change of text within the sections.

Would that work?

[pvanlaake]

Jonathan, it's not just about being able to understand the diffs or trace the provenance of certain text, but git will just give up at some stage where the individual diffs become too large. Furthermore, splitting contents into multiple files breaks the git process no matter the complexity. A new repo would avoid the problems with diffs.

On a table for the mapping, I think this is a great idea, not just for us as the CF community during the design stage but also as a reference for old-timer conventions users to jump to the relevant content based on the "old" structure. I am not at the stage where I could do this with confidence, however. Initially, I will focus on two core sections of the conventions: (1) coordinate systems (broadly ch 4, 5, 6, 7.1); and (2) description of data (ch 3, 7.2-5). As soon as I have that structure worked out (not fully implemented) I'll produce that table for review, within a few weeks after starting on the repo. Target timeline for a "public" demonstration of the new structure using these two sections: CF Workshop 2027.

More in general, I hope that others will join in with the design and the restructuring to get as many viewpoints as possible included as early as possible, specifically inviting some of the active participants in the hackathon at the recent workshop. There will also be regular general invitations to review during the restructuring.

Any further suggestions from your side would be much appreciated.

[ChrisBarker-NOAA]

All:

Workflow is always a balance between a well managed process and an low-enough friction system that work can get done.

I would argue that for a major re-structuring, the "usual" git PR from a fork, review, merge, etc process really does not work well.

In particular, @pvanlaake's proposal is a major restructuring, I think the best process would be to create a new fork of the CF repo, and use that to do this restructuring -- it would not lose any history or anything else, and it could be kept up to date (rebased) with the main one as much as we want. Then there would be a small team for that repo, with different permissions than for the main CF repo.

git is an amazing tool -- the entire point of the "distributed" model is that all repos are essentially equal and you can branch and merge between repos (almost) as easily as you can within one repo.

In particular, whatever we do -- if at least @pvanlaake can't push to the branch that hold the large refactor, this is not going to go well.

Again -- go ahead and try -- to my point above, a fork can be made at any time in the future.

[pvanlaake]

Hi Chris,

Ethan made a branch devel-semantics of main and I forked it locally, which is where I work on the restructuring. That actually works pretty well because I am only changing the structuring of the chapters - the appendices and other material (mostly) maintain the same single-file structure so my edits there will be nicely tracked. In terms of merging PRs into the branch, yes there is some friction there but nothing major so long as everybody stays committed to helping this effort along to a point where it can be properly assessed.

[ChrisBarker-NOAA]

OK, I'll shut up now and we'll see how it goes :-)