DeepSpeed

Extreme Speed and Scale for DL Training and Inference

출판물

자료 보기:

- Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
  
  Guanhua Wang, Chengming Zhang, Zheyu Shen, Ang Li, Olatunji Ruwase
  
  MSR-TR-2024-40 | September 2024
  
  글쓴이 Microsoft
  
  Publication
- Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training
  
  Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, Minjia Zhang
  
  June 2024
  
  DOI Publication Publication
- FastPersist: Accelerating Model Checkpointing in Deep Learning
  
  Guanhua Wang, Olatunji Ruwase, Bing Xie, Yuxiong He
  
  MSR-TR-2024-27 | June 2024
  
  글쓴이 Microsoft
  
  Publication
- ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
  
  Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yang, Lei Yang, Yuxiong He
  
  ICLR 2024 | May 2024
  
  Publication PDF
- ZeRO++: Extremely Efficient Collective Communication for Large Model Training
  
  Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yang, Lei Yang, Yuxiong He
  
  ICLR 2024 | March 2024
  
  Publication Publication
- DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
  
  Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, L. Song, Samyam Rajbhandari, Yuxiong He
  
  October 2023
  
  DOI Publication PDF Github
- DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
  
  Conglong Li, Zhewei Yao, Xiaoxia Wu, Minjia Zhang, Yuxiong He
  
  December 2022
  
  Publication PDF
- Extreme Compression for Pre-trained Transformers Made Simple and Efficient
  
  Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, Yuxiong He
  
  NeurIPS 2022 | November 2022
  
  Preprint
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  
  Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He (yuxhe)
  
  MSR-TR-2022-9 | June 2022
  
  글쓴이 Microsoft
  
  Publication
- DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
  
  Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
  
  MSR-TR-2022-21 | June 2022
  
  글쓴이 Microsoft
  
  Publication
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
  
  Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He
  
  ICML 2022 | January 2022
  
  Publication
- Scalable and Efficient MoE Training for Multitask Multilingual Models
  
  Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, Hany Hassan Awadalla
  
  September 2021
  
  Publication Github 프로젝트
- ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
  
  Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
  
  April 2021
  
  Publication 프로젝트
- 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB’s Convergence Speed
  
  Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He
  
  April 2021
  
  Publication 프로젝트
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  
  Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
  
  May 2020
  
  Publication 동영상 Github 프로젝트

- Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
  
  Guanhua Wang, Chengming Zhang, Zheyu Shen, Ang Li, Olatunji Ruwase
  
  MSR-TR-2024-40 | September 2024
  
  글쓴이 Microsoft
  
  Publication
- Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training
  
  Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, Minjia Zhang
  
  June 2024
  
  DOI Publication Publication
- FastPersist: Accelerating Model Checkpointing in Deep Learning
  
  Guanhua Wang, Olatunji Ruwase, Bing Xie, Yuxiong He
  
  MSR-TR-2024-27 | June 2024
  
  글쓴이 Microsoft
  
  Publication
- ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
  
  Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yang, Lei Yang, Yuxiong He
  
  ICLR 2024 | May 2024
  
  Publication PDF
- ZeRO++: Extremely Efficient Collective Communication for Large Model Training
  
  Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yang, Lei Yang, Yuxiong He
  
  ICLR 2024 | March 2024
  
  Publication Publication
- DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
  
  Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, L. Song, Samyam Rajbhandari, Yuxiong He
  
  October 2023
  
  DOI Publication PDF Github
- DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
  
  Conglong Li, Zhewei Yao, Xiaoxia Wu, Minjia Zhang, Yuxiong He
  
  December 2022
  
  Publication PDF
- Extreme Compression for Pre-trained Transformers Made Simple and Efficient
  
  Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, Yuxiong He
  
  NeurIPS 2022 | November 2022
  
  Preprint
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  
  Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He (yuxhe)
  
  MSR-TR-2022-9 | June 2022
  
  글쓴이 Microsoft
  
  Publication
- DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
  
  Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
  
  MSR-TR-2022-21 | June 2022
  
  글쓴이 Microsoft
  
  Publication
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
  
  Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He
  
  ICML 2022 | January 2022
  
  Publication
- Scalable and Efficient MoE Training for Multitask Multilingual Models
  
  Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, Hany Hassan Awadalla
  
  September 2021
  
  Publication Github 프로젝트
- ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
  
  Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
  
  April 2021
  
  Publication 프로젝트
- 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB’s Convergence Speed
  
  Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He
  
  April 2021
  
  Publication 프로젝트
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  
  Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
  
  May 2020
  
  Publication 동영상 Github 프로젝트
- Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
  
  Guanhua Wang, Chengming Zhang, Zheyu Shen, Ang Li, Olatunji Ruwase
  
  MSR-TR-2024-40 | September 2024
  
  글쓴이 Microsoft
  
  Publication
- FastPersist: Accelerating Model Checkpointing in Deep Learning
  
  Guanhua Wang, Olatunji Ruwase, Bing Xie, Yuxiong He
  
  MSR-TR-2024-27 | June 2024
  
  글쓴이 Microsoft
  
  Publication
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
  
  Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He
  
  ICML 2022 | January 2022
  
  Publication
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  
  Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
  
  May 2020
  
  Publication 동영상 Github 프로젝트
- Scalable and Efficient MoE Training for Multitask Multilingual Models
  
  Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, Hany Hassan Awadalla
  
  September 2021
  
  Publication Github 프로젝트

- Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
  
  Guanhua Wang, Chengming Zhang, Zheyu Shen, Ang Li, Olatunji Ruwase
  
  MSR-TR-2024-40 | September 2024
  
  글쓴이 Microsoft
  
  Publication
- FastPersist: Accelerating Model Checkpointing in Deep Learning
  
  Guanhua Wang, Olatunji Ruwase, Bing Xie, Yuxiong He
  
  MSR-TR-2024-27 | June 2024
  
  글쓴이 Microsoft
  
  Publication
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  
  Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He (yuxhe)
  
  MSR-TR-2022-9 | June 2022
  
  글쓴이 Microsoft
  
  Publication
- DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
  
  Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
  
  MSR-TR-2022-21 | June 2022
  
  글쓴이 Microsoft
  
  Publication
- Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training
  
  Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, Minjia Zhang
  
  June 2024
  
  DOI Publication Publication
- DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
  
  Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, L. Song, Samyam Rajbhandari, Yuxiong He
  
  October 2023
  
  DOI Publication PDF Github
- ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
  
  Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
  
  April 2021
  
  Publication 프로젝트
- 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB’s Convergence Speed
  
  Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He
  
  April 2021
  
  Publication 프로젝트
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  
  Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
  
  May 2020
  
  Publication 동영상 Github 프로젝트
- ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
  
  Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yang, Lei Yang, Yuxiong He
  
  ICLR 2024 | May 2024
  
  Publication PDF
- ZeRO++: Extremely Efficient Collective Communication for Large Model Training
  
  Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yang, Lei Yang, Yuxiong He
  
  ICLR 2024 | March 2024
  
  Publication Publication
- Extreme Compression for Pre-trained Transformers Made Simple and Efficient
  
  Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, Yuxiong He
  
  NeurIPS 2022 | November 2022
  
  Preprint
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
  
  Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He
  
  ICML 2022 | January 2022
  
  Publication
- DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
  
  Conglong Li, Zhewei Yao, Xiaoxia Wu, Minjia Zhang, Yuxiong He
  
  December 2022
  
  Publication PDF
- Scalable and Efficient MoE Training for Multitask Multilingual Models
  
  Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, Hany Hassan Awadalla
  
  September 2021
  
  Publication Github 프로젝트