🤸🏻 ODD PI 26.4 Stretch Objectives

[abarciauskas-bgse]

Description

This issue is a container for ODD's multiple stretch objectives. ODD has limited extra capacity, but we would like to capture these ideas in case other objectives are completed early or capacity is added to the team.

🧊 Stretch Objective 1: Apache Iceberg STAC backend: icestac

Why now?

• We have demonstrated that STAC Geoparquet is a really useful format for hosting STAC metadata and making it accessible to clients without imposing any burden on an API (just read the data from cloud storage with rustac/duckdb)
• There isn’t an existing way for an organization to deploy STAC Geoparquet in place of something like PgSTAC
  • How to serve/advertise the data?
  • Appending new records is not very simple
• Apache Iceberg is a well-established system for maintaining living/breathing parquet-backed data stores with
• ACID transactions (which is a critical feature for collections that are continually adding new data)
• We should build a system for maintaining a STAC catalog (multiple collections) in an Apache Iceberg store. This will require assessing the existing functionality around spatial data in Iceberg and duckdb.

Role/LOE

Questions

Stretch Objective 2: 📝 Cloud-Optimized Data Guidance

Why now?

Given our expertise and community involvement, we are well positioned to continue to develop cloud-optimized data guidance for NASA and the broader community.

Role/LOE

0.2 (<1 sprint)

Tasks/Acceptance Criteria

• Update diagram, addressing feedback on vision from ODD Team Leads Meeting - 2026/06/29 - Notes by Gemini
• Transition plan
• Share and get feedback and buy in during virtual stores co-working session
• Data lifecycle follow up from meeting: https://github.com/NASA-IMPACT/2026-odsi-retreat/issues/1
  Includes instruction on how to decide storage location and policies for icechunk stores
  Draft PR for Cloud-Optimized Data Decision Tree in guide.cloudnativegeo.org

Other stretch objectives

• 🏁Warped chunk caching
• ⚖️ Production ready multiscales pipeline
• 🚀Demonstration of flux with existing icechunk stores
• 🔥Iceberg for Fires data: 13. Outpacing the Flames: Fire Behavior Data, Delivered Fast; collaboration with data services
• Deck.gl-raster work: TBD if this will be to work on the disasters use case, protected bucket access, or CSDA footprint clustering.
• Create a more consumable (browsable? something more like 2i2c.org/roadmap?) zarr-python roadmap. Goal to be more transparent about zarr-python ongoing and future development activities.

[sharkinsspatial]

@abarciauskas-bgse For more detail on stretch objective 1 I'm copying the text from the internal labs ticket I created https://github.com/developmentseed/labs/issues/485

Geospatial Iceberg next steps

Why

It's clear that there is significant momentum around the work we've invested in geoparquet and stac-geoparquet and that parquet is an excellent metadata storage target for many of our partners and projects. It also seems that a file format by itself may not be sufficient for many of our partners' operational workflows. These operational use cases require a table format like Iceberg. There are a large number of reasons why this table layer is necessary for most cases but the most important for our purposes are likely

• ACID Transactions
• Row-Level Deletes & Updates
• Flexible Schema Evolution
• Hidden/Adaptable Partitioning

If there are questions about additional rationales, please reach out and we can discuss in more detail.

The State of Iceberg Geospatial support

Confusing 🤕 I had written some notes on this prior to my paternity leave but I think this recent Wherobots post provides a better overall explanation of the current state of the ecosystem support for native Geo types in Iceberg. Wherobots has notionally built v3 Geo support into WherobotsDB as have other big lake providers like Snowflake.

This is great for these companies and their customers but we are an open source company working to build fully open, community based solutions for our partners. To accomplish this we need open engine support for writing and querying v3 Geo types.

Where should our effort go?

At this point I think we can comfortably focus on 2 community engines as the primary options for interacting with v3 Geo types in Iceberg.

• DuckDB
• iceberg-rust

DuckDB is becoming ubiquitous and many of our potential query and writing scenarios can be accomplished on single nodes.

iceberg-rust can provide a native Iceberg interface we can use for building more bespoke/modular writing pipelines as well and as a drop-in pushdown support in Datafusion through the IcebergTableProvider.

The Iceberg geospatial support in DuckDB has been complex due to DuckDB's extension model and the extension inter-dependency complications between its spatial and iceberg extensions. Large scale efforts are underway to move GEOMETRY and GEOGRAPHY support into the DuckDB core. For a much deeper read on these efforts you can review:

• GEOMETRY Rework: Part 1 - Logical Type duckdb/duckdb#19136
• Centralizing GEOMETRY and GEOGRAPHY types paleolimbot/duckdb-geography#20

Given the low-level nature of this development and our lack of C expertise, I think progress on complete Geo/Iceberg support in DuckDB is just something we'll need to track and monitor.

So that leaves us to focus our efforts on iceberg-rust. @kylebarron has already laid most of the groundwork for this via apache/arrow-rs#8373. This comment apache/iceberg-rust#1884 (comment) outlines some of the remaining work we would need to tackle. This part is relatively straightforward. The harder part is ...

Geospatial Partition Transforms

Iceberg has excellent functionality for abstracting multiple partitioning strategies away from users via hidden partitioning. Many of our systems require multiple partitioning strategies for low-latency performance across different query modes (temporal/spatial). The bulk of the work we need to do is creating the specification for Geospatial Partition Transforms as outlined in the original proposal doc https://docs.google.com/document/d/1iVFbrRNEzZl8tDcZC81GFt01QJkLJsI9E2NBOt21IRI/edit?tab=t.0#heading=h.9q3ovqvpis89 and then in iceberg-rust:

• Build a geospatial transform variant (likely Z-curve to start).
• Build predicate projection so that spatial filters leverage the synthetic partition column.
• Partition transforms aren't very helpful without optimized sorting so we'll also need to implement data rewriting to use space filling curves.

The one benefit of approaching this via iceberg-rust is that these features will certainly take some time to land in DuckDB and it will mean that the most open solution will also be the most performant.

I don't fully grasp the scope/scale of all this work. I can step through this plan with @kylebarron and make some estimates that we can use to make a more detailed plan with more accurate levels of effort.