Repository navigation
New CrossSubjecEvaluation that supports transfer learning methods #1077
Description
Activity
I agree, this topic of transfer learning become hot and we are missing "metadata awereness" in CrossSubject.
I did something similar in pyRiemann-qiskit to use RPA pipeline, but it only covers one of the use cases you mentioned.
I see potential in this proposal.
@bruAristimunha FYI
@bruAristimunha I am preparing the PR. It will be a big one. The new Cross Subject will be fully adapted for the latest trends in Cross Subject Transfer Learning. It will allow people to fairly compare their results in Transfer Learning, which is the main purpose of MOABB - fair comparable results.
Hey @toncho11, short version: I'm in favour of adding transfer-learning splits!
My one constraint is that they go in as scikit-learn cross-validators, not as a new evaluation engine with modes. The reason is narrow and practical: the protocol/evaluation determines the ranking. If our split contract drifts from sklearn's, two results stop being comparable, and people stop trusting the leaderboard. So the design target is to add the capability without touching the contract.
Why I want to stay close to scikit-learn. Everything in
moabb/evaluations/splitters.pyis aBaseCrossValidatorthat does one thing:def split(self, y, metadata) -> Iterator[(train_idx, test_idx)]. Andbase.pyconsumes it in exactly one place:for cv_ind, (train_idx, test_idx) in enumerate(splitter.split(y, metadata)). That is the whole interface. A transfer split that yields(train_idx, test_idx)intometadata.indexinherits the existing caching, scoring, and result handling for free. No special case, no parallel evaluation class to keep in sync.The proposal. Transfer learning with calibration means: train on the source subjects, optionally add a labelled slice of the target subject, then test on the rest of the target. That is one parameter on top of
CrossSubjectSplitter, not a new protocol. The calibration slice has to be drawn per session, otherwise a single session can land entirely in train and the test score reads as cross-subject when it is really within-session. So the parameter controls per-session calibration, and the per-session draw is the part that keeps the claim honest.class CrossSubjectTransferSplitter(BaseCrossValidator): """Leave-one-subject-out, with optional labelled calibration from the target. For each held-out target subject: train = all source subjects (+ ``calibration_size`` of each target session) test = the remaining target trials calibration_size=0.0 reproduces CrossSubjectSplitter exactly (zero-calibration transfer). calibration_size=0.2 gives the "20% labelled target" setting. Calibration trials are drawn per session and stratified by class, so no session leaks wholesale into train and the test set stays balanced. """ def __init__(self, calibration_size: float = 0.0, random_state=None): self.calibration_size = calibration_size self.random_state = random_state def split(self, y, metadata): rng = check_random_state(self.random_state) all_index = metadata.index.values subjects = metadata["subject"].unique() for target in subjects: target_mask = (metadata["subject"] == target).to_numpy() source_idx = all_index[~target_mask] if self.calibration_size <= 0: yield source_idx, all_index[target_mask] continue calib_parts, test_parts = [], [] target_meta = metadata[target_mask] for session in target_meta["session"].unique(): sess_mask = (target_meta["session"] == session).to_numpy() sess_idx = all_index[target_mask][sess_mask] calib_idx, test_idx = train_test_split( sess_idx, train_size=self.calibration_size, stratify=y[target_mask][sess_mask], random_state=rng, ) calib_parts.append(calib_idx) test_parts.append(test_idx) train_idx = np.concatenate([source_idx, *calib_parts]) yield train_idx, np.concatenate(test_parts)
Usage is the plain scikit-learn loop, so it works as a standalone splitter without touching any evaluation class:
from moabb.evaluations import CrossSubjectTransferSplitter # calibration_size=0.0 is source-only (same as today's CrossSubjectSplitter); # 0.2 adds a per-session labelled slice of the target subject to train. splitter = CrossSubjectTransferSplitter(calibration_size=0.2, random_state=42) for train_idx, test_idx in splitter.split(y, metadata): ...
Wiring it into
CrossSubjectEvaluation(to reuse its caching/results/scoring) is a separate, optional follow-up, tracked in #1088. If the framing works for you, I'll open a PR with the splitter and tests againstFakeDataset, mirroring howCrossSubjectSplitteris tested, and we iterate from there.I understand the motivation for keeping the standard scikit-learn splitter contract, and I agree that this is a good fit for labelled calibration, where target calibration samples can simply be added to train_idx.
However, my current code also covers unlabeled target adaptation and trialwise source-only prediction. The unlabeled case cannot be expressed cleanly with only (train_idx, test_idx): unlabeled target samples are not labelled training data, but the estimator still needs access to them during fit/adaptation.
For trialwise prediction, a splitter could technically yield one target trial at a time as test_idx, but then the folds are no longer one fold per held-out subject. My implementation keeps the LOSO subject-level fold and only changes the prediction call so the estimator sees one test trial at a time.
In short:
- My code supports more transfer-learning cases, especially unlabeled target adaptation and trialwise prediction. This is important for methods such as Riemannian Procrustes Analysis, a common transfer-learning technique that uses only unlabeled target data.
- I tried to reuse the existing MOABB machinery as much as possible: data loading, splitting, scoring, result construction, model saving, fitted checks, and the general cross-subject structure are still based on MOABB’s existing evaluation code.
- My class inherits from CrossSubjectEvaluation and can be used independently. It does not replace the existing CrossSubjectEvaluation; it adds a separate target-aware evaluation for protocols that cannot be represented cleanly by a standard (train_idx, test_idx) splitter.
I have thought carefully about this design and tried to cover the important details. Modes are actually importatant because in an article you can put the string HOS_UNLABELED_50P and this way it is clear in which category you are competing.
I will put my current code in a PR so we can discuss the design concretely there.
@bruAristimunha @gcattan Also it might be more productive to make a Teams meeting next week and discuss it.
We can change the train_idx, test_idx to something more generic, such train_idx, valid_idx, test_idx, and the train can be empty too.
Can you point for the code for me look?
I have been working on it for the last 1 month. I just need a few more touches.
Also let's take Riemannian Procrustes Analysis (RPA) for example. RPA needs access to which data belongs to which subject. In my code I say these samples belong to this subject in the fit() method. Without this information there is no transfer learning.
The separate calibration slice added in #1093 and the explicit splitter hook in #1207 cover a useful distinction for small-budget adaptation. One concrete documentation example could be an absolute chronological budget from an early run, with evaluation restricted to later runs, rather than a fraction of every target session.
I maintain IntentLab, where a recent external-to-MOABB experiment used the first ten eligible run-04 trials for each target participant and only runs 08/12 for evaluation. Alignment used those ten recordings without labels; personal temperature scaling used their labels. Remaining run-04 trials were excluded from scoring. The protocol and trial-ID manifest are public here: https://github.com/shrut10/intentlab-bci/tree/main/experiments/adaptation-v1
That is a different question from fitting an alignment reference to the full unlabelled target block. A small example using the new splitter hook could make the budget, discarded trials, label access and run chronology inspectable together, with tests for positional indices, disjoint slices and insufficient calibration trials. I have not ported this pipeline to MOABB or run a MOABB benchmark, so this is a concrete use case for the documentation rather than a compatibility claim. Would that example be useful to the project?
I am proposing a new class called CrossSubjectTargetAwareEvaluation that extends cross-subject evaluation to pipelines that require subject metadata or optional unlabeled/labeled target-subject calibration data. The existing CrossSubjectEvaluation becomes one of 6 options that can be configured.
All this is motivated by the large number of papers on Transfer Learning. So we need:
I myself am working on a transfer learning cross subject method and all this is motivated by real needs.
Currently in my code I have defined:
This will allow people to start comparing their methods clearly. Some will focus on HOS_UNLABELED_20P, others on HOS_SOURCE_ONLY_TRIALWISE, but the results will become comparable. I work on HOS_SOURCE_ONLY_TRIALWISE.
So the objective is:
Notes:
I will provide code in a PR later. As you can imagine, it is a big feature :)
I think the best strategy is to have CrossSubjectTargetAwareEvaluation along the CrossSubjectEvaluation. And allow people to test it for a while.