Download / Extract / Reproject pipeline currently relies on the SatPy package as a convenient wrapper around xarray, rasterio, GDAL etc. However there is a limitation (bug?) in which the reprojection of satellite granules to the mercator mapping grid can only occur for a given band after the entire band has been loaded into memory. This introduces significant performance issues because it does not take advantage of the lazy operations (i.e., index / spatial subsetting) on disparate chunks.
To increase performance, should consider direct "manual" implementing the necessary SatPy operations using the underlying Xarray / Rasterio / Gdal packages after chunking and spatially subsetting the granules. These include:
- Geolocation of Arrays
- Reprojection
- Atmospheric Correction / Rayleigh Scattering
- Enhancements of Images (e.g., TrueColor RGB)
Have been reading through the SatPy source code to reproduce TrueColor image based on the underlying libraries, with the results in a JupyterLab notebook.
Download / Extract / Reproject pipeline currently relies on the SatPy package as a convenient wrapper around xarray, rasterio, GDAL etc. However there is a limitation (bug?) in which the reprojection of satellite granules to the mercator mapping grid can only occur for a given band after the entire band has been loaded into memory. This introduces significant performance issues because it does not take advantage of the lazy operations (i.e., index / spatial subsetting) on disparate chunks.
To increase performance, should consider direct "manual" implementing the necessary SatPy operations using the underlying Xarray / Rasterio / Gdal packages after chunking and spatially subsetting the granules. These include:
Have been reading through the SatPy source code to reproduce TrueColor image based on the underlying libraries, with the results in a JupyterLab notebook.