Skip to content

Add filters to compress (deflate) and chunk standard and custom netCDF outputs - #94

Merged
analytophile merged 4 commits into
mainfrom
deflate_and_chunk
Apr 13, 2026
Merged

analytophile merged 4 commits into
mainfrom
deflate_and_chunk

Conversation

@huard

@huard huard commented Apr 10, 2026

Copy link
Copy Markdown
Collaborator

Pull Request Checklist:

  • This PR addresses an already opened issue (for bug fixes / features)
  • Tests for the changes have been added (for bug fixes / features)
    • (If applicable) Documentation has been added / updated (for bug fixes / features)

What kind of change does this PR introduce?

  • Compresses NetCDF standard and custom outputs using the deflate filter.
  • Chunks NetCDF standard and custom outputs, optimizing for time series analysis (one chunk covers the entire time series).

Compression is particularly useful with simulations that do not include q_obs. Without compression, we still need to store the full array of NaNs.

Does this PR introduce a breaking change?

  • I ran the RavenPy test suite and no errors were raised.

Other information:

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds NetCDF4 compression (deflate) and chunking settings to both standard and custom NetCDF outputs, aiming to reduce file sizes (especially for NaN-heavy variables) and improve performance for time-series access.

Changes:

  • Enable NetCDF variable deflate compression (configurable via NETCDF_DEFLATE_LEVEL) across standard and custom outputs.
  • Add chunking configuration for NetCDF variables, including heuristics to target ~10MB chunks for 2D outputs.
  • Introduce a computed output time dimension size (Options.n_out_time) and a helper to approximate custom-output timestep counts.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/StandardOutput.cpp Adds deflate/chunking for time and output variables; updates NetCDF metadata helpers to apply compression/chunking.
src/RavenInclude.h Introduces NetCDF compression/chunking constants and adds optStruct::n_out_time.
src/ParseInput.cpp Computes Options.n_out_time from duration/timestep/output interval.
src/CustomOutput.h Declares ApproximateNumTimeSteps for custom NetCDF chunk sizing.
src/CustomOutput.cpp Implements timestep approximation and applies deflate/chunking to custom NetCDF variables.
src/ModelInitialize.cpp Minor whitespace-only change.
src/Makefile Adds a debug build note.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/StandardOutput.cpp Resolved
Comment thread src/StandardOutput.cpp Resolved
Comment thread src/StandardOutput.cpp Resolved
Comment thread src/ParseInput.cpp Resolved
Comment thread src/CustomOutput.cpp Resolved
@huard

huard commented Apr 13, 2026

Copy link
Copy Markdown
Collaborator Author

Running into segfaults for a large scale simulation.... investigating.

@analytophile analytophile left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great addition.

@analytophile
analytophile merged commit 4763686 into main Apr 13, 2026
15 checks passed
@analytophile
analytophile deleted the deflate_and_chunk branch April 13, 2026 19:25
@analytophile

Copy link
Copy Markdown
Collaborator

this passed the full set of benchmarking models, with output files ~1/2 to ~1/3rd of original size. No segfaults here.

@huard huard mentioned this pull request Apr 14, 2026
1 task done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants