https://answer.freetools.me/tags/deepspeed/ Recent content in DeepSpeed on Answer Hugo -- 0.152.2 zh-cn Mon, 09 Mar 2026 05:54:35 +0800 https://answer.freetools.me/%E5%8D%83%E4%BA%BF%E5%8F%82%E6%95%B0%E6%A8%A1%E5%9E%8B%E5%A6%82%E4%BD%95%E5%A1%9E%E8%BF%9B%E6%9C%89%E9%99%90%E6%98%BE%E5%8D%A1zero%E5%A6%82%E4%BD%95%E7%94%A8%E5%88%86%E7%89%87%E6%B6%88%E9%99%A4%E6%95%B0%E6%8D%AE%E5%B9%B6%E8%A1%8C%E7%9A%84%E5%86%85%E5%AD%98%E5%86%97%E4%BD%99/ Mon, 09 Mar 2026 05:54:35 +0800 https://answer.freetools.me/%E5%8D%83%E4%BA%BF%E5%8F%82%E6%95%B0%E6%A8%A1%E5%9E%8B%E5%A6%82%E4%BD%95%E5%A1%9E%E8%BF%9B%E6%9C%89%E9%99%90%E6%98%BE%E5%8D%A1zero%E5%A6%82%E4%BD%95%E7%94%A8%E5%88%86%E7%89%87%E6%B6%88%E9%99%A4%E6%95%B0%E6%8D%AE%E5%B9%B6%E8%A1%8C%E7%9A%84%E5%86%85%E5%AD%98%E5%86%97%E4%BD%99/ 深入解析ZeRO（零冗余优化器）如何通过三阶段分片技术消除数据并行的内存冗余。从混合精度训练的内存消耗分析入手，详细阐述优化器状态、梯度、参数分片的数学原理，对比ZeRO与模型并行、流水线并行的通信开销，并介绍ZeRO-Offload和ZeRO-Infinity如何突破GPU内存墙。 https://answer.freetools.me/%E6%A2%AF%E5%BA%A6%E6%A3%80%E6%9F%A5%E7%82%B9%E5%A6%82%E4%BD%95%E8%AE%A9%E5%A4%A7%E6%A8%A1%E5%9E%8B%E8%AE%AD%E7%BB%83%E7%AA%81%E7%A0%B4%E6%98%BE%E5%AD%98%E7%93%B6%E9%A2%88%E4%BB%8E%E6%97%B6%E9%97%B4%E6%8D%A2%E7%A9%BA%E9%97%B4%E5%88%B0%E9%80%89%E6%8B%A9%E6%80%A7%E9%87%8D%E8%AE%A1%E7%AE%97%E7%9A%84%E6%8A%80%E6%9C%AF%E8%BF%9B%E5%8C%96/ Mon, 09 Mar 2026 05:36:18 +0800 https://answer.freetools.me/%E6%A2%AF%E5%BA%A6%E6%A3%80%E6%9F%A5%E7%82%B9%E5%A6%82%E4%BD%95%E8%AE%A9%E5%A4%A7%E6%A8%A1%E5%9E%8B%E8%AE%AD%E7%BB%83%E7%AA%81%E7%A0%B4%E6%98%BE%E5%AD%98%E7%93%B6%E9%A2%88%E4%BB%8E%E6%97%B6%E9%97%B4%E6%8D%A2%E7%A9%BA%E9%97%B4%E5%88%B0%E9%80%89%E6%8B%A9%E6%80%A7%E9%87%8D%E8%AE%A1%E7%AE%97%E7%9A%84%E6%8A%80%E6%9C%AF%E8%BF%9B%E5%8C%96/ 深入解析Gradient Checkpointing的核心原理：从2016年Tianqi Chen的开创性论文到NVIDIA的选择性重计算技术，揭示大模型训练如何通过"用计算换内存"突破显存瓶颈。涵盖O(√n)内存复杂度的数学证明、Transformer激活内存的精确公式、Sequence Parallelism与Selective Recomputation的协同优化，以及PyTorch/DeepSpeed/Megatron-LM的实战配置。 https://answer.freetools.me/%E5%8D%83%E4%BA%BF%E5%8F%82%E6%95%B0%E6%A8%A1%E5%9E%8B%E5%A6%82%E4%BD%95%E6%8B%86%E5%88%86%E5%88%B0%E5%A4%9A%E5%BC%A0gpu%E4%B8%8A%E8%AE%AD%E7%BB%83%E4%BB%8E%E6%95%B0%E6%8D%AE%E5%B9%B6%E8%A1%8C%E5%88%B03d%E5%B9%B6%E8%A1%8C%E7%9A%84%E6%8A%80%E6%9C%AF%E7%AA%81%E5%9B%B4/ Mon, 09 Mar 2026 04:31:19 +0800 https://answer.freetools.me/%E5%8D%83%E4%BA%BF%E5%8F%82%E6%95%B0%E6%A8%A1%E5%9E%8B%E5%A6%82%E4%BD%95%E6%8B%86%E5%88%86%E5%88%B0%E5%A4%9A%E5%BC%A0gpu%E4%B8%8A%E8%AE%AD%E7%BB%83%E4%BB%8E%E6%95%B0%E6%8D%AE%E5%B9%B6%E8%A1%8C%E5%88%B03d%E5%B9%B6%E8%A1%8C%E7%9A%84%E6%8A%80%E6%9C%AF%E7%AA%81%E5%9B%B4/ 深入解析大模型分布式训练的核心技术。从数据并行的内存瓶颈，到张量并行的列/行切分策略，再到流水线并行的bubble问题与1F1B调度，最后到ZeRO与3D并行的终极方案。涵盖Megatron-LM、DeepSpeed等主流框架的实现原理，以及GPT-3等千亿参数模型的实际训练配置。