make it clear where a YAXArray is stored

[gdkrmr]

It is often not clear where the actual data of a YAXArray is stored. It would be nice to know if a cube was stored in memory, on a local path or on some remote location. In the example below there is no indicator in the printed output of the object that indicates where it is stored.

It would also be nice to have some kind of function store_path or similar that could indicate this.

julia> dc_maps[:air_temperature_2m]
┌ 1440×720×1978 YAXArray{Float32, 3} Mean Air Temperature at 2 m ┐
├────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────── dims ┐
  ↓ lon Sampled{Float64} -179.875:0.25:179.875 ForwardOrdered Regular Points,
  → lat Sampled{Float64} -89.875:0.25:89.875 ForwardOrdered Regular Points,
  ↗ time Sampled{DateTime} [DateTime("1979-01-05T00:00:00"), …, DateTime("2021-12-31T00:00:00")] ForwardOrdered Irregular Points                                                                                                                        
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any} with 27 entries:
  "original_scale_factor"     => 1.0
  "long_name"                 => "Mean Air Temperature at 2 m"
  "original_add_offset"       => 0.0
  "time_coverage_end"         => "2021-12-27T00:00:00.000000000"
  "acknowledgment"            => "ERA5 hourly data on single levels from 1959 to present"
  "geospatial_lat_resolution" => 0.25
  "time_coverage_start"       => "1979-01-01T00:00:00.000000000"
  "name"                      => "air_temperature_2m"
  "standard_name"             => "mean_air_temperature_2m"
  "reported_day"              => 5.0
  "description"               => "ERA5 Reanalysis Products"
  "processing_steps"          => Any["Merging nc files", "Resampling by daily mean", "Converting to °C from K", "Resampling b…
  "original_name"             => "t2m"
  "date_modified"             => "2022-11-04 15:41:36.233472"
  "references"                => Any["https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation"]
  "geospatial_lat_min"        => -89.875
  "temporal_resolution"       => "8D"
  "geospatial_lon_resolution" => 0.25
  "time_period"               => "8D"
  "_FillValue"                => NaN
  "units"                     => "°C"
  "project"                   => "DeepESDL"
  "geospatial_lon_max"        => 179.875
  "source"                    => Any["https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels"]
  "license"                   => "Terms and conditions of the DeepESDL data distribution"
  "geospatial_lat_max"        => 89.875
  "geospatial_lon_min"        => -179.875
├───────────────────────────────────────────────────────────────────────────────────────────────────────────── loaded lazily ┤
  data size: 7.64 GB
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘


julia> readcubedata(dc_maps[:air_temperature_2m])
┌ Warning: Loading a Cube of size 7.64 GB.
└ @ YAXArrays.Cubes ~/.julia/packages/YAXArrays/2yFST/src/Cubes/Cubes.jl:254
┌ 1440×720×1978 YAXArray{Float32, 3} Mean Air Temperature at 2 m ┐
├────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────── dims ┐
  ↓ lon Sampled{Float64} -179.875:0.25:179.875 ForwardOrdered Regular Points,
  → lat Sampled{Float64} -89.875:0.25:89.875 ForwardOrdered Regular Points,
  ↗ time Sampled{DateTime} [DateTime("1979-01-05T00:00:00"), …, DateTime("2021-12-31T00:00:00")] ForwardOrdered Irregular Points                                                                                                                        
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any} with 27 entries:
  "original_scale_factor"     => 1.0
  "long_name"                 => "Mean Air Temperature at 2 m"
  "original_add_offset"       => 0.0
  "time_coverage_end"         => "2021-12-27T00:00:00.000000000"
  "acknowledgment"            => "ERA5 hourly data on single levels from 1959 to present"
  "geospatial_lat_resolution" => 0.25
  "time_coverage_start"       => "1979-01-01T00:00:00.000000000"
  "name"                      => "air_temperature_2m"
  "standard_name"             => "mean_air_temperature_2m"
  "reported_day"              => 5.0
  "description"               => "ERA5 Reanalysis Products"
  "processing_steps"          => Any["Merging nc files", "Resampling by daily mean", "Converting to °C from K", "Resampling b…
  "original_name"             => "t2m"
  "date_modified"             => "2022-11-04 15:41:36.233472"
  "references"                => Any["https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation"]
  "geospatial_lat_min"        => -89.875
  "temporal_resolution"       => "8D"
  "geospatial_lon_resolution" => 0.25
  "time_period"               => "8D"
  "_FillValue"                => NaN
  "units"                     => "°C"
  "project"                   => "DeepESDL"
  "geospatial_lon_max"        => 179.875
  "source"                    => Any["https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels"]
  "license"                   => "Terms and conditions of the DeepESDL data distribution"
  "geospatial_lat_max"        => 89.875
  "geospatial_lon_min"        => -179.875
├────────────────────────────────────────────────────────────────────────────────────────────────────────── loaded in memory ┤
  data size: 7.64 GB
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

[felixcremer]

On the right side of the last box just before the data size there is either a "loaded lazily" or a "loaded in memory" to indicate this. We could change the wording if you think there is more useful info to include.

There is a loadingstatus function which dispatches currently on DiskArrays to indicate that it is loaded lazily, but we could also dispatch on more information.

https://github.com/JuliaDataCubes/YAXArrays.jl/blob/main/src/Cubes/Cubes.jl#L530-L531

[gdkrmr]

did not notice, thanks!

For reference, YAXArrays.Cubes.loadingstatus(x) returns a string "loaded into memory" or "loaded lazily".

It would be nice to have an official interface function to query this, else feel free to close the issue.

[felixcremer]

I would say the official interface function for this is isdisk from DiskArrays since all of this is defined there.