Skip to content

README.zh.md

Latest commit

 

History

History
266 lines (197 loc) · 33.8 KB

README.zh.md

File metadata and controls

266 lines (197 loc) · 33.8 KB
Raw

License Apache 2.0 PyPI version Downloads Build OpenSSF Best Practices Twitter Japanese Twitter Chinese Zhihu Slack

English · 简体中文

开发者交流会 (Office Hours)

DeepSpeed 于每月最后一个星期二美东时间 12:00（北京时间次日凌晨 00:00 / 01:00）定期举行线上 Office Hours，共同讨论开发规划、新特性与技术设计等。该会议对所有人公开，欢迎任何人加入并交流提问。 会议基于 Zoom 进行，可通过此处链接直接参会。

最新动态

查看更多历史动态

极致速度与超大规模深度学习训练

DeepSpeed 成功支撑了全球众多极具影响力的大规模语言模型训练，例如 MT-530B 以及 BLOOM。DeepSpeed 汇聚了一系列前沿的系统级创新，使超大规模深度学习训练变得既可行又极其高效，大幅提升了开发易用性，并在模型可扩展性维度彻底重新定义了深度学习的训练边界。这些突破性技术包括 ZeRO、ZeRO-Infinity、3D 并行（3D-Parallelism）、Ulysses 序列并行、DeepSpeed-MoE 等。

DeepSpeed 生态应用

DeepSpeed 是微软 AI at Scale 核心战略的重要支柱，旨在规模化赋能下一代人工智能前沿能力，更多背景可查阅此处详情。

DeepSpeed 已被广泛应用于训练各类超大规模模型，以下为部分知名代表案例（若您希望在此列出您的模型，欢迎提交 PR）：

DeepSpeed 已无缝集成至众多主流开源深度学习框架与生态：

文档指引
在 Transformers 中使用 DeepSpeed
在 Accelerate 中使用 DeepSpeed
在 Lightning 中使用 DeepSpeed
在 MosaicML 中使用 DeepSpeed
在 Determined 中使用 DeepSpeed
在 MMEngine 中使用 DeepSpeed

构建流水线状态

平台 / 运行环境 持续集成状态
NVIDIA nv-pre-compile-ops modal-torch-latest
AMD amd-mi200
CPU torch-latest-cpu
Intel Gaudi hpu-gaudi2
Intel XPU xpu-max1100
生态集成 aws-accelerate
其他检查 Formatting pages-build-deployment Documentation Statuspython
华为昇腾 NPU Huawei Ascend NPU

安装指南

通过 pip 安装是快速上手 DeepSpeed 最便捷的方式，这将安装 DeepSpeed 的最新稳定发布版（且无需强制绑定特定的 PyTorch 或 CUDA 版本）。DeepSpeed 包含多个我们通常称为“算子 (Ops)”的 C++/CUDA 原生扩展。默认情况下，所有这些扩展/算子都会通过依赖 ninja 的 PyTorch JIT C++ 扩展加载器在运行时按需即时（JIT）编译并动态链接。

环境要求

  • 在安装 DeepSpeed 之前必须预先安装 PyTorch。
  • 为获得完整的特性支持，建议使用 PyTorch >= 2.0 版本，最佳实践为使用最新的 PyTorch 稳定发布版。
  • 需具备用于编译 C++/CUDA/HIP 扩展的 CUDA 或 ROCm 编译器，例如 nvcc 或 hipcc。
  • 我们日常重点开发与测试的 GPU 架构如下。若您的 GPU 未列在其中并不意味着无法运行，而是指 DeepSpeed 在以下硬件平台上经过了最充分的测试验证：
    • NVIDIA：Pascal、Volta、Ampere 以及 Hopper 架构
    • AMD：MI100 与 MI200

社区贡献硬件支持

  • DeepSpeed 目前已支持多种异构硬件加速器：
贡献方 硬件设备 加速器标识 (Accelerator Name) 贡献方已验证 上游官方已验证
华为 (Huawei) 华为昇腾 NPU (Huawei Ascend NPU) npu 是 (Yes) 否 (No)
英特尔 (Intel) Intel(R) Gaudi(R) 2 AI 加速器 hpu 是 (Yes) 是 (Yes)
英特尔 (Intel) Intel(R) 至强(R) 处理器 (Xeon Processors) cpu 是 (Yes) 是 (Yes)
英特尔 (Intel) Intel(R) 数据中心 GPU Max 系列 (Data Center GPU Max) xpu 是 (Yes) 是 (Yes)
太初元碁 (Tecorigin) SDAA sdaa 是 (Yes) 否 (No)

PyPI 安装

我们定期向 PyPI 发布最新版本，在绝大多数场景下推荐直接从 PyPI 安装：

pip install deepspeed

安装完成后，您可以通过运行 DeepSpeed 环境诊断报告来验证安装，并查看当前机器所兼容支持的扩展/算子列表：

ds_report

如果您希望预先编译安装任意 DeepSpeed 算子/扩展（而非在运行时 JIT 编译），或通过 PyPI 获取预编译好的算子，请参阅我们的进阶安装指南。

Windows 安装支持

DeepSpeed 在 Windows 平台上对训练和推理均已支持大量核心功能。关于此特性的详细背景可参阅官方博客。目前在 Windows 上尚未支持的特性包括异步 I/O (AIO) 与 GDS（其本身不支持 Windows）。

  1. 安装 PyTorch（例如 pytorch 2.3+cu121）。
  2. 安装 Visual C++ 生成工具（例如 VS2022 C++ x64/x86 生成工具）。
  3. 以管理员权限打开 Cmd 控制台以创建所需的符号链接目录，并确保 MSVC 相关工具已添加至 PATH 环境变量中；或者以管理员权限启动 Visual Studio 2022 的 Developer Command Prompt。
  4. 运行 build_win.bat，即可在 dist 目录下构建 Wheel 安装包。

延伸阅读

所有关于 DeepSpeed 的官方文档、教程指南与技术博客均可在我们的官方网站上找到：deepspeed.ai

描述
快速上手 (Getting Started) DeepSpeed 初步入门指南
JSON 配置详解 (DeepSpeed JSON Configuration) DeepSpeed 核心配置选项
API 文档 (API Documentation) DeepSpeed API 自动生成文档
教程指南 (Tutorials) 官方系列技术实战教程
技术博客 (Blogs) 官方前沿技术深度博文

持续集成 (CI) 硬件赞助

作为一个开源项目，我们依赖合作伙伴为持续集成（CI）流水线提供硬件计算资源。目前，Modal 慷慨地为我们资助了 GPU CI 运行所需的全部硬件。Modal 是一个专注于模型推理、微调与批处理任务的一站式 AI 基础设施平台。现在访问 https://modal.com 即可获取每月 30 美元的免费额度开启体验。DeepSpeed 团队得到了来自 Modal 团队全方位的鼎力支持，在此向您的业务强烈推荐 Modal 服务。

参与贡献

DeepSpeed 非常欢迎开源社区的贡献！关于代码格式化规范、单元测试运行等详细信息，请参阅我们的贡献指南。
由衷感谢所有为 DeepSpeed 做出卓越贡献的开发者们！

开发者原创性声明 (DCO)

本项目欢迎任何建议与代码贡献。对于绝大多数贡献，您需要同意开发者原创性声明 (DCO)，声明您认可发布于 https://developercertificate.org 的对应条款适用于该次特定贡献，并拥有授予相关贡献权限的合法权利。

DCO 针对每次 Git Commit 生效，因此每个提交都需要附加签署声明（Sign-off）。您只需在提交代码时添加 -s 参数即可完成签署（例如 git commit -s -m "Commit message"）。在 Pull Request 界面中，也可以直接点击 DCO 检查项完成授权。

行为准则

本项目遵循 Microsoft 开源行为准则。欲了解更多信息，请查阅行为准则常见问题，如有其他疑问或意见，亦可直接联系 opencode@microsoft.com。

代表性学术论文

  1. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He. (2019) ZeRO: memory optimizations toward training trillion parameter models. arXiv:1910.02054 and In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20).

  2. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. (2020) DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial).

  3. Minjia Zhang, Yuxiong He. (2020) Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping. arXiv:2010.13369 and NeurIPS 2020.

  4. Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He. (2021) ZeRO-Offload: Democratizing Billion-Scale Model Training. arXiv:2101.06840 and USENIX ATC 2021. [paper] [slides] [blog]

  5. Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He. (2021) 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed. arXiv:2102.02888 and ICML 2021.

  6. Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He. (2021) ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. arXiv:2104.07857 and SC 2021. [paper] [slides] [blog]

  7. Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He. (2021) 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed. arXiv:2104.06069 and HiPC 2022.

  8. Conglong Li, Minjia Zhang, Yuxiong He. (2021) The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models. arXiv:2108.06084 and NeurIPS 2022.

  9. Yucheng Lu, Conglong Li, Minjia Zhang, Christopher De Sa, Yuxiong He. (2022) Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam. arXiv:2202.06009.

  10. Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He. (2022) DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale arXiv:2201.05596 and ICML 2022. [pdf] [slides] [blog]

  11. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, Bryan Catanzaro. (2022) Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model arXiv:2201.11990.

  12. Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, Yuxiong He. (2022) Extreme Compression for Pre-trained Transformers Made Simple and Efficient. arXiv:2206.01859 and NeurIPS 2022.

  13. Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He. (2022) ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers. arXiv:2206.01861 and NeurIPS 2022 [slides] [blog]

  14. Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He. (2022) DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. arXiv:2207.00032 and SC 2022. [paper] [slides] [blog]

  15. Zhewei Yao, Xiaoxia Wu, Conglong Li, Connor Holmes, Minjia Zhang, Cheng Li, Yuxiong He. (2022) Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers. arXiv:2211.11586.

  16. Conglong Li, Zhewei Yao, Xiaoxia Wu, Minjia Zhang, Yuxiong He. (2022) DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing. arXiv:2212.03597 ENLSP2023 Workshop at NeurIPS2023

  17. Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, Yuxiong He. (2023) Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases. arXiv:2301.12017 and ICML2023.

  18. Syed Zawad, Cheng Li, Zhewei Yao, Elton Zheng, Yuxiong He, Feng Yan. (2023) DySR: Adaptive Super-Resolution via Algorithm and System Co-design. ICLR:2023.

  19. Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, Yuxiong He. (2023) Scaling Vision-Language Models with Sparse Mixture of Experts. arXiv:2303.07226 and Finding at EMNLP2023.

  20. Quentin Anthony, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He, Aamir Shafi, Mustafa Abduljabbar, Hari Subramoni, Dhabaleswar Panda. (2023) MCR-DL: Mix-and-Match Communication Runtime for Deep Learning arXiv:2303.08374 and will appear at IPDPS 2023.

  21. Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, Abhinav Bhatele. (2023) A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training arXiv:2303.06318 and ICS 2023.

  22. Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He. (2023) ZeRO++: Extremely Efficient Collective Communication for Giant Model Training arXiv:2306.10209 and ML for Sys Workshop at NeurIPS2023 [blog]

  23. Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, Yuxiong He. (2023) ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation arXiv:2303.08302 and ENLSP2023 Workshop at NeurIPS2023 [slides]

  24. Pareesa Ameneh Golnari, Zhewei Yao, Yuxiong He. (2023) Selective Guidance: Are All the Denoising Steps of Guided Diffusion Important? arXiv:2305.09847

  25. Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, Yuxiong He. (2023) DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales arXiv:2308.01320.

  26. Xiaoxia Wu, Zhewei Yao, Yuxiong He. (2023) ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats arXiv:2307.09782 and ENLSP2023 Workshop at NeurIPS2023 [slides]

  27. Zhewei Yao, Xiaoxia Wu, Conglong Li, Minjia Zhang, Heyang Qin, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He. (2023) DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention arXiv:2309.14327

  28. Shuaiwen Leon Song, Bonnie Kruft, Minjia Zhang, Conglong Li, Shiyang Chen, Chengming Zhang, Masahiro Tanaka, Xiaoxia Wu, Jeff Rasley, Ammar Ahmad Awan, Connor Holmes, Martin Cai, Adam Ghanem, Zhongzhu Zhou, Yuxiong He, et al. (2023) DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies arXiv:2310.04610 [blog]

  29. Zhewei Yao, Reza Yazdani Aminabadi, Stephen Youn, Xiaoxia Wu, Elton Zheng, Yuxiong He. (2023) ZeroQuant-HERO: Hardware-Enhanced Robust Optimized Post-Training Quantization Framework for W8A8 Transformers arXiv:2310.17723

  30. Xiaoxia Wu, Haojun Xia, Stephen Youn, Zhen Zheng, Shiyang Chen, Arash Bakhtiari, Michael Wyatt, Reza Yazdani Aminabadi, Yuxiong He, Olatunji Ruwase, Leon Song, Zhewei Yao (2023) ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks arXiv:2312.08583

  31. Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, Shuaiwen Leon Song. (2024) FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design arXiv:2401.14112

  32. Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminadabi, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He. (2024) System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

  33. Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, Minjia Zhang. (2024) Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training arXiv:2406.18820

  34. Stas Bekman, Samyam Rajbhandari, Michael Wyatt, Jeff Rasley, Tunji Ruwase, Zhewei Yao, Aurick Qiao, Yuxiong He. (2025) Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences arXiv:2506.13996

  35. Tingfeng Lan, Yusen Wu, Bin Ma, Zhaoyuan Su, Rui Yang, Tekin Bicer, Masahiro Tanaka, Olatunji Ruwase, Dong Li, Yue Cheng. (2025) ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updates arXiv:2505.12242

  36. Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder.(2025) Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems EMNLP 2025

  37. Xinyu Lian, Masahiro Tanaka, Olatunji Ruwase, Minjia Zhang. (2026) SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips arxiv, ASPLOS 2026

视频讲座与教程

  1. DeepSpeed KDD 2020 专题教程
    1. 概览与架构 (Overview)
    2. ZeRO 优化器与超大模型训练 (ZeRO + large model training)
    3. 17B 参数 T-NLG 演示 (17B T-NLG demo)
    4. 极速 BERT 训练与 RScan 调优 (Fastest BERT training + RScan tuning)
    5. DeepSpeed 实战深度拆解：第一部分 (part 1)、第二部分 (part 2)、第三部分 (part 3)
    6. 常见问题解答 (FAQ)
  2. 微软研究院网络研讨会 (Microsoft Research Webinar)
  3. 在 AzureML 上运行 DeepSpeed (DeepSpeed on AzureML)
  4. 基于 DeepSpeed 的大模型训练与推理 // Samyam Rajbhandari // LLMs in Prod 大会演讲 [演示胶片]
  5. 社区优质教程

💡 文档维护说明：本中文文档由社区志愿者（@JasonYeYuhe）翻译维护，最后同步更新于 2026年09月27日。如发现内容与官方英文原版存在差异或新特性滞后，欢迎提交 PR 共同完善！