FIX keep CrossSubject results subject-specific for grouped folds

[lindicaphxag-tech]

Summary

CrossSubjectEvaluation supports grouped cross-validation strategies where a single test fold can contain more than one subject (for example GroupKFold(n_splits < n_subjects)). The flattened evaluation engine currently assigns the entire fold to test_meta["subject"].iloc[0] and then only splits scores by session.

That silently mixes different held-out subjects that share a session label and stores the results under whichever subject appears first.

Context in the evaluation-safety series

This is the CrossSubject follow-up to #1207 and #1210. #1207 enables explicit grouped splitters; #1210 preserves session identity for multi-session folds; this PR completes the same result-provenance invariant across subjects so grouped CrossSubject folds cannot silently attribute several held-out subjects to the first one.

Fix

Keep one fit per CV fold, but score and store the fitted model separately for every held-out subject/session that still needs this pipeline.

• add an opt-in per-subject score grouping contract to the shared evaluation engine;
• enable it only for CrossSubjectEvaluation;
• preserve partial-cache work plans so already-computed subjects are not recomputed;
• when save_model=True, store the one fitted fold model under every held-out subject that owns a result;
• default leave-one-subject-out behavior is unchanged.

Regression coverage

A focused test uses four subjects, two sessions, and n_splits=2, so every GroupKFold test fold holds out two subjects simultaneously. It verifies:

• all four subjects remain present in the results;
• each subject/session pair appears exactly once;
• all eight expected result rows are emitted;
• n_samples_test matches the true trial count of each subject/session pair.

This is the CrossSubject analogue of the same provenance invariant addressed for multi-session CrossSession folds in #1210.

[lindicaphxag-tech]

Exact-head validation is green on cf46c4a0a9bbb28de880596d12ae21122b5ae0ad: the focused multi-subject CrossSubject provenance regression passes and repository pre-commit passes on the touched evaluation files. Evidence: https://github.com/lindicaphxag-tech/lindicaphxag-tech/actions/runs/37145822074. I’ll hold scope here for review.

[lindicaphxag-tech]

Updated exact-head validation: on 92897f6fd460298f02da7510df0ffc312ae00436, current-head pre-commit and the multi-subject provenance/partial-work-plan regressions both pass. This supersedes the earlier evidence SHA after the branch cleanup/squash. Evidence: https://github.com/lindicaphxag-tech/lindicaphxag-tech/actions/runs/37183338993. I’ll hold this head for review.

[lindicaphxag-tech]

@codex review

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[bruAristimunha]

Thanks @lindicaphxag-tech. The provenance invariant for grouped CrossSubject folds is the right one, and the design keeps one fit per fold while scoring/storing per held-out subject, consistent with #1210. The default LeaveOneGroupOut path is equivalent because every fold still holds one subject, so score_subjects=[subject] collapses to the pre-PR grouping. CI is green on 92897f6.

One ask before I approve: the description commits to preserving partial-cache work plans — could you add a focused test that pre-fills the result cache for one of the held-out subjects of a fold and asserts the fold is still fit once and only the uncached subject gets a new row? That makes the contract load-bearing.

Merge-order note for the series: #1206 → #1210 → #1207 → #1211 minimises hand-merging; this one touches _evaluate_fold and _build_task_list and should land last, so expect a rebase once the others are in.

[lindicaphxag-tech]

Addressed on current head b72b2db1: I added test_cross_subject_partial_cache_fits_remaining_fold_once. It pre-populates the result cache for one subject in the first two-subject held-out fold (and both subjects in the other fold), then runs process() with a counting estimator. The regression asserts the grouped fold is fitted exactly once, the cached subject receives no new row, and only the uncached subject gains its per-session rows. pre-commit.ci is green on this head. I’ll keep #1211 last in the merge order as requested and rebase after #1206/#1207 land.

[bruAristimunha]

@lindicaphxag-tech #1206, #1207 and #1210 are now on develop, so this one has a real conflict in moabb/evaluations/base.py (the _evaluate_fold save-path call and _build_task_list), not just whats_new.rst. Could you rebase onto current develop? The partial-cache regression you added is what I asked for, so once CI is green on the rebased head this is ready to approve. Thanks.

[lindicaphxag-tech]

Rebased the PR onto current develop (d7e0951) and replayed only the grouped CrossSubject provenance changes. The partial-cache regression you requested is retained, and the branch is mergeable again. CI is re-running on head 3d4f28d.

[bruAristimunha]

Thanks for the rebase, @lindicaphxag-tech. I merged current develop into your branch: _evaluate_fold now passes paradigm/suffix (#1206) inside your per-subject save loop, _build_task_list keeps the _validate_test_fold_metadata call (#1207), and test_cross_subject_multisubject_fold_preserves_provenance looks for the models under the paradigm-namespaced directory that #1206 introduced. 16 relevant tests pass locally; merging when CI is green.

[bruAristimunha]

Correction: your rebase (f6e7bdc) already carried the #1206/#1207 integration and the namespaced model-path test — my push was rejected as stale and nothing of mine landed. Checked the rebased head locally (16 relevant tests pass); merging on green.

[lindicaphxag-tech]

The only remaining CI failure is Docs, and its traceback is unrelated to this PR: plot_dreyer_clf_scores_vs_subj_info.py hit an OSF download 429 Too Many Requests. Test, Test-braindecode, What's New, and Link Check are green. I don't have Actions write permission to rerun the failed job from the fork.