Add automated notebook testing with Papermill #602
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces a GitHub Actions workflow to run and validate Jupyter notebooks in CI using a new runner script. Adds scripts to mock PyMC sampling for faster execution, updates test dependencies to include papermill, and documents the notebook runner usage. Also updates the interrogate badge to reflect new coverage.
drbenvincent
added
documentation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces automated testing for Jupyter notebooks in CI using Papermill. The implementation includes a runner script that mocks PyMC's MCMC sampling with faster prior predictive sampling to validate notebooks execute without errors.
Key changes:
- New notebook runner script with filtering capabilities for different notebook types
- Mock PyMC sampling implementation that replaces expensive MCMC with prior predictive sampling (10 draws)
- GitHub Actions workflow that runs notebooks in parallel across three categories (PyMC, sklearn, and other notebooks)
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/run_notebooks/runner.py | Main script for executing notebooks with Papermill, includes filtering and logging |
| scripts/run_notebooks/injected.py | Mock implementation of pm.sample that uses prior predictive sampling |
| scripts/run_notebooks/README.md | Documentation for the notebook runner usage and CI integration |
| .github/workflows/test_notebook.yml | GitHub Actions workflow for parallel notebook testing |
| pyproject.toml | Adds papermill to test dependencies |
| docs/source/_static/interrogate_badge.svg | Updates documentation coverage badge from 96.3% to 96.0% |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #602 +/- ##
==========================================
+ Coverage 93.74% 93.77% +0.02%
==========================================
Files 41 41
Lines 6827 6827
Branches 458 458
==========================================
+ Hits 6400 6402 +2
+ Misses 267 266 -1
+ Partials 160 159 -1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Updated the mock for pm.sample() to use 100 draws instead of 50 for prior predictive sampling, as reflected in both the injected script and documentation. This change aims to provide more robust validation during notebook execution.
Updated the mock for pm.sample to use 500 draws instead of 100 to ensure compatibility with notebook code that iterates over posterior samples, such as plot_ate which defaults to 500 draws. Adjusted documentation and injected.py accordingly.
Introduces skip_notebooks.yml to specify notebooks incompatible with prior predictive sampling mock. Updates runner.py to filter out these notebooks and reduces MIN_DRAWS from 500 to 100 for faster execution.
Replaces import of LinearRegression from causalpy.skl_models with sklearn's LinearRegression and removes execution count from the first code cell.
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Replaces hardcoded sample size with dynamic calculation based on the length of 'uncertainty' to prevent errors when fewer than 500 samples are available. Also resets execution count to null for the affected notebook cell.
Expanded the skip_notebooks.yml file to include iv_pymc.ipynb, which requires JAX not available in the CI test environment. Updated comments to clarify reasons for skipping each notebook.
Installs Graphviz as a system dependency in the test_notebook GitHub Actions workflow to support notebooks or tests that require it.
Collaborator
Author
|
passing! @NathanielF See |
Collaborator
Author
|
bugbot review |
PR SummaryIntroduces CI to validate docs notebooks execute without errors.
Written by Cursor Bugbot for commit c15a229. This will update automatically on new commits. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces a GitHub Actions workflow to run and validate Jupyter notebooks in CI using a new runner script. Adds scripts to mock PyMC sampling for faster execution, updates test dependencies to include papermill, and documents the notebook runner usage. Also updates the interrogate badge to reflect new coverage.
📚 Documentation preview 📚: https://causalpy--602.org.readthedocs.build/en/602/