Quantization strategy / alignment with pending code

[fmigneault]

Hi @tik65536

I am working on a feature that would allow automatically calculating the aggregated zone data from finer levels provided in a collection provider configuration to produce coarser zones levels requested by the API.

The idea is that if a collection provider only contains a dataset of, eg: L15 zones, it could return any L0-L15 zones, even when the L0-L14 ones are not in the stored dataset. The feature would be able to also support many sub-groups (eg: L0, L5, L10, L15) to offer "checkpoints" where requested data would aggregate based on the closest higher-level the collection providers offers. When there are a lot of fine-resolution zones, that should help reduce the amount of storage necessary for all other coarser levels.

To implement this feature, I need to invoke a function that does the finer->coarser quantization step within the get_data of the various AbstractCollectionProvider, since the higher level zone Ids must be fetched to get a match, and then quantized to coarser zones.

In other words, some kind of "groupby(finer->coarser)" call of the result dataframe after each of the following:

pydggsapi/pydggsapi/dependencies/collections_providers/clickhouse_collection_provider.py

Lines 86 to 90 in e4f0e6b

	try:
	db_result = self.db.execute(query, {'cellid_list': zoneIds}, with_column_types=True)
	except Exception as e:
	logger.error(f'{__name__} get_data failed : {e}')
	raise Exception(f'{__name__} get_data failed : {e}')

pydggsapi/pydggsapi/dependencies/collections_providers/parquet_collection_provider.py

Lines 89 to 93 in e4f0e6b

	try:
	result_df = datasource.conn.sql(sql, params=[zoneIds]).df()
	except Exception as e:
	logger.error(f'{__name__} {datasource_id} query data error: {e}')
	raise Exception(f'{__name__} {datasource_id} query data error: {e}')

pydggsapi/pydggsapi/dependencies/collections_providers/zarr_collection_provider.py

Lines 88 to 99 in e4f0e6b

	sql = f"""select {cols} from ds where ("{id_col}" in ({', '.join(f"'{z}'" for z in zoneIds)})) and ({cql_sql}) """
	zarr_result = xr.Dataset.from_dataframe(ctx.sql(sql).to_pandas().set_index(id_col))
	else:
	cols = set(datatree.data_vars) if ("*" in datasource.data_cols) else set(datasource.data_cols)
	cols = list(cols - set(datasource.exclude_data_cols))
	idx_mask = datatree[id_col].isin(np.array(zoneIds, dtype=datatree[id_col].dtype))
	zarr_result = datatree.sel({id_col: idx_mask})
	zarr_result = zarr_result.to_dataset()[cols]
	except Exception as e:
	# Zarr will raise exception if nothing matched
	logger.error(f'{__name__} {datasource_id} sel failed: {e}')
	return result

Since I am aware you are working on https://github.com/LandscapeGeoinformatics/pydggsapi/compare/alternative_dggrs_representations with notable fixes for datetime support in Zarr, I would like to synchronize to facilitate the PR merges/avoid conflicts with this feature.
However, because of the datetime-per-depth naming that creates repeated temporal dimensions (#65 (comment)), I cannot merge it directly, as I believe it defies the concept of datetime dimension in the first place.

Do you think the unique datetime could be applied in the Zarr provider?

[fmigneault]

This is what I have so far: main...crim-ca:pydggsapi:quantized-subzones

[tik65536]

To implement this feature, I need to invoke a function that does the finer->coarser quantization step within the get_data of the various AbstractCollectionProvider, since the higher level zone Ids must be fetched to get a match, and then quantized to coarser zones.
In other words, some kind of "groupby(finer->coarser)" call of the result dataframe after each of the following:

Shall we pass the dggrs instance to the get_data function to generate the zone IDs at the finest refinement level of the collections with the input zone ID? Otherwise, it doesn't know which zones should be queried.

Alternatively, we can extend the get_data function with an additional parameter, groupby_zoneIds, whose length is assumed to be the same as the' zoneIds' list. It provides a 1-to-1 mapping of zone IDs to the zone ID at the coarser level.

At the api endpoint logic, if the requested refinement level isn't supported by the collections, the API should generate the corresponding zone IDs at the finest refinement level, given the requested zone ID. Then, it performs the get_data query with the newly generated zone IDs as zoneIds and the original requested zone IDs as groupby_zoneIds.

The collection provider should perform aggregation/quantisation if the groupby_zoneIds is presented, and return the data with zone IDs from the groupby_zoneIds.

However, because of the datetime-per-depth naming that creates repeated temporal dimensions (#65 (comment)), I cannot merge it directly, as I believe it defies the concept of datetime dimension in the first place.

Yes, I plan to do it later this week, after the data conversion is complete. To return only one temporal dimension for zone data retrieval with multiple zone depths.

Do you think the unique datetime could be applied in the Zarr provider?

However, I believe it is related to the implementation of the zone data retrieval endpoint, rather than the Zarr provider. Or are you referring to other things?

Thx, Tik

[fmigneault]

Alternatively, we can extend the get_data function with an additional parameter, groupby_zoneIds

That is essentially the method I took in main...crim-ca:pydggsapi:quantized-subzones

New parameters are:

        quantize_zones_mapping: Dict[str, List[str]] = None,
        quantize_property_methods: ZoneDataPropertyQuantizationMethod = None,

A new quantize_zones method is defined to perform the quantization within get_data if the parameters are provided/not-empty.

It provides a 1-to-1 mapping of zone IDs to the zone ID at the coarser level.

That was my first iteration, but I realized that depending on the DGGRS, the children might be shared between more than one parent since zones are not fully contained (eg: IGEO7 has 1 center, 6 11/12 area-overlap immediate neighbours, and 6 1/12 area following neighbours).

Instead, I take the Dict[str, List[str]] to allow N-1 aggregation mapping. I do not perform any fancy weighted aggregation yet.

Yes, I plan to do it later this week, after the data conversion is complete.

Excellent! Thanks

Zarr provider

Indeed the zarr collection provider, not the endpoint response.
I was specifically referring to the diffs in pydggsapi/dependencies/collections_providers/zarr_collection_provider.py between your/my branch, size we are both doing some adjustments to the get_data.

Normally, my quantize_zones call simply inserts itself just before your adjustments to handle the datetime dimension, so the merge should be transparent, but we would need to validate just in case. I had to do a zarr->pandas->quantize-pandas->zarr workaround to support an aggregation feature that xarray does not directly handle.

[tik65536]

That was my first iteration, but I realized that depending on the DGGRS, the children might be shared between more than one parent since zones are not fully contained (eg: IGEO7 has 1 center, 6 11/12 area-overlap immediate neighbours, and 6 1/12 area following neighbours).

That's the difference between spatial children and hierarchical children; IGEO7 got exactly one parent if you use the hierarchical one.
That is still debate on which one should be used.

The quantisation implementation is amazing 😍

I am reviewing the merge failures; it appears that they are mostly due to the padding parameter and some minor ordering issues.

[tik65536]

However, because of the datetime-per-depth naming that creates repeated temporal dimensions (#65 (comment)), I cannot merge it directly, as I believe it defies the concept of datetime dimension in the first place.

For this, I have a temporary fix to return a single datetime dimension with the deepest zone level. 5134a89

[fmigneault]

That's the difference between spatial children and hierarchical children; IGEO7 got exactly one parent if you use the hierarchical one. That is still debate on which one should be used.

In this case (for quantization), the issue is more about finding children than parents, because the feature assumes that children zones would be available to aggregate into the parent/coarser zones.

But yes, I agree that depending on the use case, the weighted aggregation is not necessary relevant (eg: finding the max value regardless of how much/weighted the area overlaps to test a "worst case" flood scenario). Doing the weighting might actually deteriorate the result in some cases by over-smoothing some extreme values on barely-overlapping children, or could be much more convoluted to interpret in other DGGRS.

Maybe the user could even have control over the aggregation strategy (eg: via a ?zone-quantization=<prop-name>(method) or similar, so they can adapt it to their use case.

[tik65536]

@fmigneault, Shall I merge the quantize_zones branch into the alternative_dggrs_representations branch or we should merge it to main ?

[fmigneault]

It can wait. It is not entirely ready.

[tik65536]

@fmigneault, Sorry for combining those fixes into one branch. I want to address the issue before it worsens (actually, it already is, 😰, sorry again).

I would like to first rename the alternative_dggrs_representations branch to fix_zarr_provider_with_temporal_support, then merge it into the main. This branch mainly fixed/enhanced:

• zarr collection provider to support the temporal dimension
• fixed zarr+zip return with temporal dimension as coordinate
• added the padding parameter to get_data to control if the return data should be padded with the input zoneIds list
  (Should I change the name to input_zoneIds_padding ?)
• IGEO7 provider support collections with zone ID in the hexstring and the integer representation
• fixed ZoneIdRepresentationType handling
• fixed tiles-api.

Then, I will create two branches: one for the zone ID representation(#58) and the other for zone data retrieval handling, including both temporal and non-temporal collections (#65).

What do you think?

Thanks, Tik

[fmigneault]

Yes, sounds good.