Skip to content

Fix brier_class() case weight normalization - #645

Open
taekop wants to merge 1 commit into
tidymodels:mainfrom
taekop:fix/brier-class-weight-normalization
Open

taekop wants to merge 1 commit into
tidymodels:mainfrom
taekop:fix/brier-class-weight-normalization

Conversation

@taekop

@taekop taekop commented Sep 29, 2026

Copy link
Copy Markdown

Closes #633.

brier_ind() normalized case_weights with a softmax (exp(w - max(w)) / sum(exp(w - max(w)))) before computing the weighted mean. This is only equivalent to a plain weighted mean when all the weights are equal; once weights are spread out, exp() distorts their relative sizes and can underflow small weights to zero, silently dropping those observations. The fix removes the softmax and normalizes by the plain sum of weights instead, matching how other weighted metrics like mn_log_loss() handle case_weights.

Since ranked_prob_score() calls the same brier_ind() helper, its weighted calculations were affected too and are fixed by the same change; I updated its case-weight test's expected value, which had been generated from the buggy output rather than the stated orf::rps() reference.

Added a test comparing brier_class() with widely spread case_weights against the same data expanded with tidyr::uncount().

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

unexpected results with weighted brier_class

1 participant