Add bias (mean signed error) metric - #737
Open
AshNicolus wants to merge 4 commits into
Open
AshNicolus wants to merge 4 commits into
AshNicolus wants to merge 4 commits into
Conversation
mse/wmse and mae/wmae are magnitude-only, so RMSE/MAE alone can't distinguish a systematic bias from random scatter - standard practice pairs them with bias (mean signed error) for this. Follows the existing mae signature/reduction pattern exactly and is auto-logged during --eval test alongside mse/mae, through the existing metric-name-agnostic aggregation path (no sqrt special-case triggers on 'bias', and linear rescale-by-state_std is already correct for non-squared metrics). Closes mllam#736
Contributor
Author
|
@sadamov could you approve the workflow run / take a look when you have a chance? |
Collaborator
|
Thanks @AshNicolus, the implementation is clean. Since this adds a metric to the default eval loop for every user and touches the metric surface that #597 (torchmetrics) will reshape, I would like to bring it to the next dev meeting before merging. Are you still interested in getting this merged once v0.7.0 is released? |
Contributor
Author
|
Yes, I'm still interested in getting this merged after v0.7.0. Looking forward to the dev meeting discussion regarding the torchmetrics integration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
neural_lam/metrics.pyhas a tight, mirrored family of magnitude-only error metrics (mse/wmse,mae/wmae), all sharing the same signature and the samemask_and_reduce_metricreduction helper. Standard NWP/verification practice pairs magnitude metrics like RMSE/MAE with bias (mean signed error,mean(pred - target)) to detect systematic over/under-forecasting - something RMSE/MAE alone can't reveal, since they can't distinguish a systematic bias from random scatter.biasdoesn't exist anywhere in the codebase under any name.Added
bias, following the exactmaesignature/reduction pattern (nopred_std-weighted variant needed, same asmae), registered it inDEFINED_METRICS, and added it toForecasterModule.test_step's existing("mse", "mae")auto-eval loop so it's computed and logged during--eval testfor free, through the already metric-name-agnosticaggregate_and_plot_metrics/create_metric_log_dictpath (verified: the"mse" in metric_namesqrt special-case doesn't false-trigger on"bias", and the linear* state_stdrescale is already the correct treatment for a non-squared metric, same as howmaeis handled today).Added
tests/test_metrics.pywith direct unit tests forbias(signed-error correctness, reduction behavior, registry lookup). Ran the fulltests/test_training.py+tests/test_plotting.pyintegration suites (27 tests, excluding network-dependentmdpcases) to confirm addingbiasto the test-metrics loop doesn't break anything - all pass.Issue Link
closes #736
Type of change
Checklist before requesting a review
pullwith--rebaseoption if possible).