Table Convention

• UUID: d517ea5f-92bb-4df1-8bbf-db736286a307
• Name: "table"
• Namespace: "table:"
• Schema URL: "https://raw.githubusercontent.com/R-CFe/zarr_convention_table/main/schema.json"
• Spec URL: "https://github.com/R-CF/zarr_convention_table/blob/main/README.md"
• Scope: Group, Array
• Extension Maturity Classification: Proposal
• Owner: @pvanlaake

Description

This Zarr Convention describes how to store tabular (row/column) data as a Zarr group, where each column of the table is stored as a separate, row-aligned Zarr array (or, where a column's structure genuinely requires it, a small Zarr sub-group — see encoding-type). Group-level properties that register the table and point to its columns use the table: namespace prefix, at the root attributes level of the group. Per-column properties, described on each column's own node, are unprefixed — see Namespacing for details.

Unlike row-oriented tabular formats, this convention is columnar by design: each column has its own Zarr data type, its own codecs, and can be read independently of the others. This makes it a natural fit for analytical/columnar access patterns (comparable to Arrow, Parquet, or a columnar database), rather than for record-by-record transactional access.

Prior Art

This convention draws on two existing efforts that independently arrived at closely related solutions.

The AnnData on-disk format has, for several years, stored tabular annotation data (DataFrame, obs/var) in Zarr and HDF5 as a group of row-aligned column arrays, with a named index array and per-column type-tagging via encoding-type attributes (categorical, nullable-integer, and related variants) — validating the core storage model at production scale, well beyond its origin domain of single-cell genomics.

The STAC table extension separately defines a table:columns list of column objects (name, type, description) plus a table:primary_geometry pointer, for describing tabular assets (Parquet, CSV, and similar) referenced from a STAC Item or Collection — but it describes such assets from the outside, as opaque files with accompanying metadata, rather than defining how the tabular data itself is laid out at the storage layer.

The offset-array structure used for linestring and polygon columns has a third source: CF Discrete Sampling Geometries' ragged array representations, which solve the same problem (a variable number of elements per instance) inside netCDF's fixed-shape-array data model via count/index variables — this convention keeps the underlying offset-array idea but drops the representation choices (contiguous vs. indexed ragged, cf_role) that exist only to compensate for netCDF lacking a native ragged-array or relational construct, since Zarr groups don't have that limitation.

This convention adopts AnnData's column-per-node storage model, including its choice to leave column-level attributes unprefixed — safe here specifically because a node is only interpreted as one of this convention's columns by being named in the namespaced table:columns entry of a group that has registered this convention.

This convention namespaces only its group-level entry-point properties (table:columns, table:index, table:primary_geometry), where that collision risk is real, and otherwise reuses the STAC table extension's field names and semantics wherever the two overlap, and adapts CF DSG's ragged-array reasoning to a columnar geometry encoding modeled on GeoArrow rather than on netCDF's ragged representations.

Scope

This convention describes a single table as one Zarr group containing one node (array or sub-group) per column, plus the metadata needed to interpret that group as a table. It supports point, linestring, and polygon geometry columns natively (see encoding-type). It does not define:

• A catalog mechanism for multiple tables (out of scope; compose with your own hierarchy or a cataloging convention as needed).
• Coordinate reference system information for geometry columns (out of scope for this convention; compose with the proj convention as needed).
• Mixed-geometry-type columns, or MultiPoint/MultiLineString/MultiPolygon (deferred; see Future Work).

Composability

• A table's row axis is an ordinary Zarr dimension and may be named via dimension_names for use with other conventions that expect named dimensions.
• This convention does not disallow non-tabular data stored alongside a table group; other conventions may be composed as needed. Added arrays or sub-groups to a table group is discouraged, however.

Motivation

• Type fidelity: a single Zarr array can only have one data type. Tabular data routinely mixes types across columns; representing each column as its own node gives every column its native, correctly-typed storage instead of forcing a lowest-common-denominator encoding.
• Columnar access: consumers can read one column without touching the others, and can choose per-column codecs suited to that column's data (e.g. a floating-point column compressed differently from a string column).
• Minimal structural metadata: the convention only records what cannot be recovered from the arrays themselves (column-to-node mapping, optional row index, per-column encoding). Anything derivable from array metadata (row count, chunk layout) is deliberately left out to avoid data that can silently go stale.
• Open column vocabulary, plain arrays by default: per-column encoding types are not a closed enum, so new column kinds can be added over time without a breaking change to the convention — but the vocabulary favors a plain array wherever the data's own shape allows it, including for a fixed number of values per row, reserving sub-group structure for cases where it is actually beneficial or required (see encoding-type), rather than defaulting to a nested structure for convenience.

Convention Registration

The convention must be registered in zarr_conventions on the table group:

{
  "zarr_conventions": [
    {
      "schema_url": "https://raw.githubusercontent.com/R-CF/zarr_convention_table/main/schema.json",
      "spec_url": "https://github.com/R-CF/zarr_convention_table/blob/main/README.md",
      "uuid": "d517ea5f-92bb-4df1-8bbf-db736286a307",
      "name": "table",
      "description": "Tabular data as a group of row-aligned column arrays"
    }
  ]
}

Applicable To

• Group — the table itself; carries table:columns, table:index, and table:primary_geometry.
• Array — each simple column; carries type, description, encoding-type, and encoding-specific properties.
• Group (nested) — each compound column (categorical, nullable, linestring, polygon); carries the same column-level properties as an array, plus its fixed structural children.

Structural Requirements

These requirements are not expressed as attributes; they constrain how the group and its child nodes must be laid out for the metadata above to be interpretable.

1. Row axis. Axis 0 of every row-aligned column node is the row axis, regardless of the column's dimensionality. A scalar-per-row column has shape (N,); a fixed multi-valued column (e.g. a point geometry column, or a station's 24 hourly readings per day) has shape (N, K, ...). Axis 0 MUST correspond to the same N rows across every row-aligned node in the table.
2. Chunk alignment. The chunk size along axis 0 MUST be identical across every node in the table whose axis-0 length equals N. This guarantees that chunk index i denotes the same set of rows in every such node, so a row-filtered read can walk chunk i across all of them in lockstep. Nodes whose axis-0 length is not N — a categorical column's categories, a ragged geometry column's coords, ring_offsets, or geom_offsets — are exempt from this requirement and may be chunked as convenient; note that geom_offsets has length N+1, not N, and is therefore exempt too even though it is row-indexed.
3. Compound columns are sub-groups. A column whose encoding-type requires more than one array (categorical, nullable, linestring, polygon) is stored as a Zarr group (not an array) containing its constituent arrays as unprefixed, positionally-fixed children. This is reserved for encodings where no single dense array can represent the column without either padding or ambiguity — see the rationale under each encoding below. A fixed number of values per row is, on its own, not such a case (see encoding-type).

CAUTION

Zarr provides no mechanism to ensure that the fidelity of the table is maintained: all groups and arrays are typically modifiable by any tool that can manipulate Zarr stores and nodes. It is up to the application or data producer to provide the necessary security, such as read-only access for unprivileged users to the Zarr store or the nodes that make up a table via file system permissions. Writing to the table, the Zarr groups and arrays, should respect the provisions of this convention.

Properties

Group-level properties

Property	Type	Description	Required	Reference
table:columns	[Column Object]	Ordered list of columns and the node storing each	Yes	table:columns
table:index	string	Name of the column (from table:columns) used as the row index/key	No	table:index
table:primary_geometry	string	Name of the column (from table:columns) that is the primary geometry column	No	table:primary_geometry

Column properties

Unprefixed — see Namespacing.

Property	Type	Description	Required	Reference
type	string	Data type of the table column (Zarr v.3 data type string)	No	type
description	string	Human-readable description of the column	No	description
encoding-type	string	How the column is physically encoded (default "array")	Conditional	encoding-type
missing_value	any	Sentinel value denoting a missing entry (lightweight nullable)	No	encoding-type
categories	array	Inline list of category values (lightweight categorical)	No	encoding-type
ordered	boolean	Whether categories are ordered	Conditional	encoding-type
axes	string[2..4]	Coordinate axis order for a geometry column, e.g. ["x", "y"]	Conditional	encoding-type

Additional Properties

Additional properties are allowed, both at the group level and on individual columns, for domain-specific metadata not covered by this convention.

Namespacing

Group-level properties (table:columns, table:index, table:primary_geometry) are namespaced because they sit directly in a group's attributes object alongside whatever other conventions or ad-hoc metadata that group may carry. Column-level properties (type, description, encoding-type, and the rest) are deliberately left unprefixed: a node is only interpreted as a column of this table by being named in the table:columns list, so its role is already established before any of its attributes are read — there is no ambiguity to resolve in the way there would be for a bare attributes-level marker. This does mean a column node's unprefixed attributes could still collide with an unrelated convention's unprefixed attributes on that same node; if that is a concern for a given deployment, prefer storing columns as arrays or sub-groups that carry no other convention's attributes.

table:columns

Ordered list of Column Objects describing each column in the table.

• Type: [Column Object]
• Required: Yes

Column Object

Field Name	Type	Description	Required
name	string	The column name, as used by consumers of the table.	Yes
node	string	The name of the child array or group (relative to the table group) storing this column.	Yes

name and node are deliberately separate: the column name is the logical, user-facing identifier and may contain characters that are inconvenient as a Zarr node name; node is the actual storage location. Column properties beyond name-to-node mapping (type, description, encoding) are attributes on the column node itself, not embedded in this list — this keeps a single source of truth for each piece of metadata.

Every node value MUST correspond to an existing child of the table group. Every child of the table group that is part of the table MUST be listed in table:columns.

table:index

Name of the column, from table:columns, whose values serve as the row index or row label.

• Type: string
• Required: No

When absent, the table has no designated row-identity column and readers SHOULD treat rows as identified positionally (0, 1, 2, ..., N-1), without materializing a synthetic index array. This is a deliberate difference from formats that always materialize a row-label array: when a table has no meaningful key, nothing should be stored to represent one.

table:primary_geometry

Name of the column, from table:columns, that is the table's primary geometry column.

• Type: string
• Required: No

Only meaningful when at least one column has a geometry encoding-type (point, linestring, or polygon). Mirrors the STAC table extension's table:primary_geometry field of the same name and purpose, for tables with more than one geometry column.

type

Data type of the table column.

• Type: string
• Required: No

Where practical, use the underlying Zarr data type identifier for the column's storage array (for a compound column, the data type of its row-aligned data array — codes for categorical, values for nullable, the coordinate array for a geometry encoding). This mirrors the recommendation in the STAC table extension to reuse the host format's own type system rather than defining a separate one.

type describes the table column's logical data type. It is independent of encoding-type, which describes how that data is physically stored in a Zarr array (see below); a categorical column might have type: "string" while its encoding-type is "categorical".

description

Human-readable description of the column.

• Type: string
• Required: No

CommonMark syntax MAY be used for rich text representation, consistent with the STAC table extension's column object.

encoding-type

How the column is physically encoded.

• Type: string
• Required: Conditional
• Default: "array"

This is an open, extensible vocabulary: implementations encountering an unrecognized encoding-type SHOULD treat the column as opaque (skip it or expose it as raw storage) rather than fail.

The vocabulary is organized in two tiers. The first tier stays a plain array and adds only descriptive attributes; the second tier requires a sub-group because no single dense array can represent the column's data without padding or ambiguity. Prefer the first tier wherever it applies — reach for the second only when the specific condition noted for that encoding actually holds.

Tier 1 — plain array, encoding-type: "array" (the default):

• Plain column. No further attributes needed.
• Nullable via sentinel (missing_value attribute set). Use when the column's real value range leaves a safe sentinel available — NaN for floats, a reserved out-of-range integer, and so on. The array holds that sentinel at missing positions; no separate mask.
• Categorical via inline list (categories attribute set, an array of the distinct values). Use for low-cardinality categoricals whose category list is small and JSON-serializable. The array itself holds integer codes indexing into categories (or, for very low cardinality, the values directly). ordered MAY be set to declare whether the categories are ordered; unordered if absent.
• Fixed multi-valued column. Any column where every row holds a fixed number of values — not just point geometry, but e.g. a station's 24 hourly observations per day, or 12 monthly values per year — is a plain dense array of shape (N, K, ...), not a sub-group; the Tier 2 encodings below exist for variable per-row cardinality or an explicit validity mask, not merely for "more than one value per row." Use dimension_names on the array to label what axis 1 (and beyond) represents, e.g. ["row", "hour"]. This convention does not itself define a way to store the per-position values (hour labels, month names) — that is a coordinate-labeling concern outside its scope.
• "point" geometry is the one reserved encoding-type value within this pattern: it stays a plain array of shape (N, D), but requires the axes attribute (below) declaring coordinate axis order from a constrained vocabulary, since geometry consumers need that role information more precisely than a free-form dimension_names label provides.

Tier 2 — sub-group, required when the condition below holds:

• "nullable". Required when no sentinel is safe in the column's value range (a column using its full representable range, or a boolean/enum column with no spare state), or when explicit Arrow/pandas-style validity-bitmap semantics are required for interoperability. Sub-group children:

  • values: an array holding the column's values (including placeholder values at masked positions). Row-aligned (length N).
  • mask: a boolean array, same shape as values, true where the corresponding value is missing. Row-aligned (length N).

• "categorical". Required when the category set is too large to reasonably live in attribute text, or contains values that don't serialize cleanly as JSON. Sub-group children:

  • categories: an array of the distinct category values. NOT row-aligned (length = number of distinct categories).
  • codes: an integer array where each value is the zero-based index into categories, or -1 to denote a missing value. Row-aligned (length N).

  Group attribute ordered (boolean, required for this encoding): whether the categories have a meaningful order. Caution: the categories/codes pairing is not enforced by Zarr — nothing prevents categories from being independently rewritten, compacted, or reordered by store maintenance that has no awareness codes depends on it positionally, since the two are ordinary, separately-addressable Zarr arrays. Treat the pair as a single atomic unit, written and rewritten together by the same process every time. Where the category list may be maintained independently of the table itself (e.g. a shared species or taxonomy list updated on its own schedule), prefer a value-based lookup instead: store a stable key in the table's column and resolve it against a separate table (composed alongside this one, joined by that key) rather than against positional index. The failure modes differ in a way that matters: a broken positional reference resolves to some value silently, with nothing to catch in a null check; a broken value-based reference fails to find its key, which is detectable. Neither is enforced by the storage layer — Zarr has no constraint-checking engine underneath it — but the value-based form fails safely and the positional form does not.

• "linestring". Required unconditionally for this geometry type: the number of vertices varies per row, so there is no fixed-multiplicity array shape that represents it without either padding every row to the longest one or an offset array. Sub-group children:

  • coords: coordinate array, shape (total_vertices, D). NOT row-aligned.
  • geom_offsets: integer array, length N+1. Row i's vertices are coords[geom_offsets[i]:geom_offsets[i+1]]. Row-indexed but length N+1, so exempt from the N-length chunk-alignment rule.

  Group attribute axes (required for this encoding): axis order of the coordinate array's last dimension, e.g. ["x", "y"] or ["x", "y", "z"].

• "polygon". Required for the same reason as linestring, with one further level of nesting for rings. Sub-group children:

  • coords: coordinate array, shape (total_vertices, D). NOT row-aligned.
  • ring_offsets: integer array, length (total_rings + 1), mapping a ring index to its vertex range in coords.
  • geom_offsets: integer array, length N+1, mapping row i to its range of ring indices in ring_offsets. The first ring in a row's range is the exterior ring; any subsequent rings are holes.

  Group attribute axes (required for this encoding), same meaning as for linestring.

categories, codes, values, mask, coords, ring_offsets, and geom_offsets are positionally-fixed children of an already-tagged compound-column group, not free-floating attributes.

axes

Coordinate axis order for a geometry column.

• Type: string[2..4], values from ["x", "y", "z", "m"]
• Required: Conditional — required when encoding-type is "point", "linestring", or "polygon"; not used otherwise.

Names the meaning of each position along the coordinate axis (the last dimension of point's array, or of linestring/polygon's coords array). This is geometry-specific and uses a constrained vocabulary, unlike the general fixed-multi-valued-column case above, which uses dimension_names for free-form axis labeling instead — geometry consumers need x/y/z/m role information specifically, not an arbitrary label.

Examples

Minimal table, no index

weather_stations/            (group; table: convention)
├── station_id                 (array, string)
├── elevation_m                (array, float32)
└── observed_at                (array, int64)

{
  "table:columns": [
    { "name": "station_id", "node": "station_id" },
    { "name": "elevation_m", "node": "elevation_m" },
    { "name": "observed_at", "node": "observed_at" }
  ]
}

station_id array attributes:

{ "type": "string", "description": "WMO station identifier" }

Table with index and lightweight nullable/categorical columns

survey_plots/                (group; table: convention)
├── plot_id                    (array, string)      -- designated index
├── land_cover                 (array, int8)         -- categorical via inline list
└── canopy_pct                 (array, float32)      -- nullable via sentinel

{
  "table:columns": [
    { "name": "plot_id", "node": "plot_id" },
    { "name": "land_cover", "node": "land_cover" },
    { "name": "canopy_pct", "node": "canopy_pct" }
  ],
  "table:index": "plot_id"
}

land_cover array attributes:

{
  "type": "string",
  "description": "Dominant land cover class",
  "categories": ["forest", "cropland", "grassland", "wetland", "urban"],
  "ordered": false
}

canopy_pct array attributes:

{
  "type": "float32",
  "description": "Canopy cover, percent (NaN where not surveyed)",
  "missing_value": "NaN"
}

Table with heavyweight nullable/categorical columns

Used here because species has thousands of possible values (too large for an inline list) and depth_m uses its full representable float range with no safe sentinel.

forest_inventory/             (group; table: convention)
├── plot_id                     (array, string, shape (N,))
├── species                     (group; categorical)
│   ├── categories                (array, string, shape (~4000,))  -- lookup table, NOT row-aligned
│   └── codes                     (array, int16, shape (N,))       -- row-aligned; the millions of plot rows live here
└── depth_m                     (group; nullable)
    ├── values                     (array, float32, shape (N,))
    └── mask                       (array, bool, shape (N,))

N is in the millions here; categories stays small regardless of table size, since its length is the number of distinct species, not the row count.

species group attributes:

{ "type": "string", "encoding-type": "categorical", "ordered": false }

depth_m group attributes:

{ "type": "float32", "encoding-type": "nullable" }

Table with a fixed multi-valued (non-geometry) column

24 hourly readings per row, no sub-group needed — same reasoning as point geometry, without the geometry framing.

station_hourly/               (group; table: convention)
├── station_id                  (array, string)       -- designated index
└── temperature_c                (array, float32, shape (N, 24), dimension_names ["row", "hour"])

{
  "table:columns": [
    { "name": "station_id", "node": "station_id" },
    { "name": "temperature_c", "node": "temperature_c" }
  ],
  "table:index": "station_id"
}

temperature_c array attributes:

{ "type": "float32", "description": "Hourly temperature readings" }

CF DSG-style trajectory table, with a LineString geometry column

The motivating case for the linestring encoding: a set of trajectories (e.g. CF's trajectory featureType), each with a variable number of vertices.

trajectories/                 (group; table: convention)
├── trajectory_id                (array, string)      -- designated index
├── platform                     (array, string)      -- inline-categorical
└── track                        (group; linestring)  -- primary geometry
    ├── coords                     (array, float64, shape (total_vertices, 2))
    └── geom_offsets                (array, int64, shape (N+1,))

{
  "table:columns": [
    { "name": "trajectory_id", "node": "trajectory_id" },
    { "name": "platform", "node": "platform" },
    { "name": "track", "node": "track" }
  ],
  "table:index": "trajectory_id",
  "table:primary_geometry": "track"
}

track group attributes:

{
  "type": "float64",
  "description": "Trajectory vertices, in temporal order",
  "encoding-type": "linestring",
  "axes": ["x", "y"]
}

Station table with a Point geometry column

Fixed multiplicity — no sub-group needed.

stations/                     (group; table: convention)
├── station_id                  (array, string)       -- designated index
└── location                    (array, float64, shape (N, 2))  -- primary geometry

location array attributes:

{ "type": "float64", "encoding-type": "point", "axes": ["x", "y"] }

Future Work

The following are intentionally deferred from this version of the convention:

• MultiPoint, MultiLineString, MultiPolygon. The same offset-nesting pattern used for linestring/polygon extends naturally by adding one further offset level (a part_offsets array between the row index and the ring/vertex level). Reserved encoding-type names, not yet specified.
• Mixed-geometry-type columns. A column that is genuinely not one consistent geometry type. The expected approach, matching the precedent set by GeoParquet's GeoArrow-native encoding (which has the same single-type-per-column restriction), is a WKB-blob fallback encoding for this case specifically.
• Coordinate reference system metadata for geometry columns — out of scope for this convention; expected to be supplied by composing with the proj convention.
• Compatibility crosswalk documents (e.g. against the AnnData DataFrame on-disk encoding, or the STAC table extension) are not included in v1. The group-level attribute names in this convention are namespaced and therefore not wire-compatible with either; a crosswalk may be published separately once the convention has stabilized.

FAQ

Why isn't row count a property?

It is fully recoverable from the shape of any row-aligned column node (or from the index column, if table:index is set). Storing it separately would create a value that could silently disagree with the arrays it is supposed to describe.

Why do linestring and polygon require a sub-group, but nullable and categorical usually don't?

For nullable/categorical, the row count is always N — a sentinel value or an inline category list can express the column's structure inside a plain array of length N, and the sub-group form is only needed when that array-level metadata becomes impractical (very large category sets, no safe sentinel). For linestring/polygon, the number of vertices per row varies, so there is no value of N-length array that represents a row's geometry without either padding to the longest row (wasteful) or an offset array pointing into a shared, non-row-aligned coordinate buffer. The offset array is not an optional convenience there — it's the only representation that is both exact and non-wasteful. A fixed (not variable) number of values per row, in contrast, never needs a sub-group at all — see encoding-type.

Why doesn't chunk alignment apply to geom_offsets?

geom_offsets has length N+1, not N, because it stores boundaries between rows rather than one value per row. The chunk-alignment guarantee (read chunk i on every column and get the same rows) is defined in terms of N-length arrays; an N+1-length array can't share that chunk grid by construction. Readers needing row i's geometry read geom_offsets[i] and geom_offsets[i+1] (a cheap, small lookup — offset arrays are typically small enough to read in full) and use those to slice into coords, rather than relying on chunk-position correspondence.

How does this relate to the AnnData DataFrame on-disk encoding, or the STAC table extension?

See Prior Art. Where this convention overlaps with STAC, it reuses STAC's field names and semantics: table:columns mirrors STAC's list of column objects shape; type/description mirror STAC's type/ description column object fields; table:primary_geometry is adopted directly. This convention adds node to the column object (STAC has no equivalent, since STAC does not describe a Zarr group of per-column nodes) and adds table:index (STAC has no row-index concept). table:row_count from the STAC extension is not part of this convention (see above).

Why does encoding-type live on the column, and table:columns only hold name/array?

To keep a single source of truth for each piece of metadata. If column type or encoding information were duplicated into the group's table:columns list and onto the column node's own attributes, the two could drift out of sync. Keeping table:columns purely structural (which node backs which column name) and all descriptive/encoding metadata on the column itself avoids that.

Acknowledgements

The template is based on the STAC extensions template via the zarr-conventions template.

Column encoding for categorical and nullable data follows the AnnData on-disk format DataFrame specification. Field naming for table:columns/type/description/ table:primary_geometry follows the STAC table extension. The linestring/polygon offset-array encoding is informed by CF Discrete Sampling Geometries and by the GeoArrow columnar geometry memory layout.