High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models

By Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liangchen Luo and 41 more. 2021

49 citations recorded by the source corpus.

Subfields: Recommender Systems, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Recommender Systems

1. Bilge Acun
2. Carole-Jean Wu
3. Geet Sethi

In ML Systems & Hardware

1. Bilge Acun
2. Christos Kozyrakis
3. Jinliang Wei
4. Marcello Maggioni
5. Shibo Wang
6. Yuanzhong Xu
7. Carole-Jean Wu
8. Blake A. Hechtman

Citations in the corpus

• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023, background): “XX models [18, 32] can reach beyond 1 trillion parameters, replete with rapidly growing dense layer components.”
• Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models (2022, methodology): “Scaling up the model size and also the amount of compute used for training has seen dramatic benefits across multiple application fields, such as natural language processing [12, 30, 46], recommendation [32], auto speech”
• RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, methodology): “This has led to a hybrid data-and model-parallel training approach (Figure 2).”