Mkphuthi/training helpers [WIP]

[mkphuthi]

Summary

Major changes:

• feature 1: Added some scripts to train and finetune models with flexibility to change datasets, schedulers and optimizers.

  • I have found that finetuning can be unstable with the default CosineAnnealingLR scheduler, so this flexibility allows using LR warmup and another scheduler such as ReduceLROnPlateau. Other best practices can be integrated as we find them
  • The training script is driven entirely by a config yaml/json with documented examples provided
  • The trainers load either pymatgen jsons or ASE-readable files and allows the user to specify the stress unit so conversion is done internally

• fix 1: The scheduler was stepping twice per epoch, this is the only change that affects the original source. It does not affect trained models but does point out that every model that has been trained so far has actually been stepped for half as many epochs as the reported number

Todos

• Right now the training script is run by directly calling them from the scripts directory. Perhaps a cleaner implementation would be to integrate with the cli so you can call "mgl train config.yaml" or "mgl finetune config.yaml" but this requires a more significant refactor

Checklist

• Train a FP to test

[shyuep]

This is fine, but don't we already have a MGLPotentialTrainer? I suggest we work from that and add the config.yaml and CLI.

[shyuep]

🤖 Automated PR review by Claude (posted via @shyuep). Based on static analysis of the diff; no code was executed locally. Note: PR is marked [WIP], so this is preliminary.

Config-driven training scripts + LR-scheduler fix. Well-structured and thoroughly documented. Main comments:

• ✅ The LR-scheduler double-step fix is real and correct. Returning ([opt],[sched]) while also stepping in on_train_epoch_end did advance the schedule twice/epoch. Moving to the explicit lr_scheduler config (interval/frequency, monitor only for ReduceLROnPlateau) is the right fix.
• ⚠️ Backward-compat / behavior change for all existing users, not just the new scripts: with the double-step removed, default CosineAnnealingLR(T_max=decay_steps) now spans decay_steps epochs instead of decay_steps/2. Retrained models will differ. The changelog documents this and gives the decay_steps/2 workaround — good — but please call it out prominently in release notes since it silently changes existing training runs.
• 🔹 Per @shyuep's comment: reconcile with the existing MGLPotentialTrainer before adding a parallel training path — worth confirming these scripts build on it rather than duplicating dataset/trainer plumbing.
• 🔹 lr_scheduler_monitor default "val_Total_Loss" only matters for ReduceLROnPlateau; confirm that metric key is actually logged during validation, else plateau scheduling will error.
• 🔹 No CI checks are reported on this branch — get lint + tests green (incl. the new tests/scripts/train/ and tests/utils/test_training.py) before this leaves WIP.

Nice work overall; the scheduler bug fix alone is valuable.