Skip to content

Add bias (mean signed error) metric - #737

Open
AshNicolus wants to merge 4 commits into
mllam:mainfrom
AshNicolus:feat/bias-metric
Open

AshNicolus wants to merge 4 commits into
mllam:mainfrom
AshNicolus:feat/bias-metric

Conversation

@AshNicolus

Copy link
Copy Markdown
Contributor

Describe your changes

neural_lam/metrics.py has a tight, mirrored family of magnitude-only error metrics (mse/wmse, mae/wmae), all sharing the same signature and the same mask_and_reduce_metric reduction helper. Standard NWP/verification practice pairs magnitude metrics like RMSE/MAE with bias (mean signed error, mean(pred - target)) to detect systematic over/under-forecasting - something RMSE/MAE alone can't reveal, since they can't distinguish a systematic bias from random scatter. bias doesn't exist anywhere in the codebase under any name.

Added bias, following the exact mae signature/reduction pattern (no pred_std-weighted variant needed, same as mae), registered it in DEFINED_METRICS, and added it to ForecasterModule.test_step's existing ("mse", "mae") auto-eval loop so it's computed and logged during --eval test for free, through the already metric-name-agnostic aggregate_and_plot_metrics/create_metric_log_dict path (verified: the "mse" in metric_name sqrt special-case doesn't false-trigger on "bias", and the linear * state_std rescale is already the correct treatment for a non-squared metric, same as how mae is handled today).

Added tests/test_metrics.py with direct unit tests for bias (signed-error correctness, reduction behavior, registry lookup). Ran the full tests/test_training.py + tests/test_plotting.py integration suites (27 tests, excluding network-dependent mdp cases) to confirm adding bias to the test-metrics loop doesn't break anything - all pass.

Issue Link

closes #736

Type of change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📖 Documentation (Addition or improvements to documentation)

Checklist before requesting a review

  • My branch is up-to-date with the target branch - if not update your fork with the changes from the target branch (use pull with --rebase option if possible).
  • I have performed a self-review of my code
  • For any new/modified functions/classes I have added docstrings that clearly describe its purpose, expected inputs and returned values
  • I have placed in-line comments to clarify the intent of any hard-to-understand passages of my code
  • I have updated the README to cover introduced code changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have given the PR a name that clearly describes the change, written in imperative form (context).
  • I have requested a reviewer and an assignee (assignee is responsible for merging). This applies only if you have write access to the repo, otherwise feel free to tag a maintainer to add a reviewer and assignee.

mse/wmse and mae/wmae are magnitude-only, so RMSE/MAE alone can't
distinguish a systematic bias from random scatter - standard practice
pairs them with bias (mean signed error) for this. Follows the
existing mae signature/reduction pattern exactly and is auto-logged
during --eval test alongside mse/mae, through the existing
metric-name-agnostic aggregation path (no sqrt special-case triggers
on 'bias', and linear rescale-by-state_std is already correct for
non-squared metrics).

Closes mllam#736
@AshNicolus

Copy link
Copy Markdown
Contributor Author

@sadamov could you approve the workflow run / take a look when you have a chance?

@sadamov

sadamov commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Thanks @AshNicolus, the implementation is clean. Since this adds a metric to the default eval loop for every user and touches the metric surface that #597 (torchmetrics) will reshape, I would like to bring it to the next dev meeting before merging. Are you still interested in getting this merged once v0.7.0 is released?

@AshNicolus

Copy link
Copy Markdown
Contributor Author

Yes, I'm still interested in getting this merged after v0.7.0. Looking forward to the dev meeting discussion regarding the torchmetrics integration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add bias (mean signed error) metric

2 participants