Fix asymmetric loss gradient disconnection

[d-ulker]

Implement robust training monitoring and fix Asymmetric Loss gradient disconnection to prevent unexpected training stalls.

The previous Asymmetric Loss implementation used disable_torch_grad_focal_loss=True, which inadvertently created a torch.no_grad() context, blocking gradient flow and preventing the model from learning. This PR corrects the loss function and introduces a ProgressMonitorCallback to detect training stalls, manage disk space, and improve NCCL communication, ensuring more reliable and recoverable multi-GPU training runs.

---

[cursor]

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}

[sourcery-ai]

Hi @uelkerd! 👋

Your private repo does not have access to Sourcery.

Please upgrade to continue using Sourcery ✨

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}