Scale MLPerf-0.6 models on Google TPU-v3 Pods
By Sameer Kumar, Victor Bitorff, Dehao Chen, Chi-Ning Chou, Blake A. Hechtman, HyoukJoong Lee, Naveen Kumar, Peter Mattson, Shibo Wang, Tao Wang and 2 more. 2019
43 citations recorded by the source corpus.
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models. MLPerf defines a suite of models, datasets and rules to follow when benchmarking to ensure results are comparable across hardware, frameworks and companies. Using this suite of models, we discuss the optimizations and techniques including choice of optimizer, spatial partitioning and weight update sharding necessary to scale to 1024 TPU chips. Furthermore, we identify properties of models that make scaling them challenging, such as limited data parallelism and unscaled weights. These optimizations contribute to record performance in transformer, Resnet-50 and SSD in the Google MLPerf-0.6 submission.
Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
- Matei A. Zaharia
- Alvaro Sanchez-Gonzalez
- Jure Leskovec
- Peter W. Battaglia
- Tobias Pfaff
- Jonathan Godwin
- Sven Gowal
- D. Narayanan
In Computer Vision and Pattern Recognition
- Matei A. Zaharia
- Luc Van Gool
- Bryan Catanzaro
- Peter W. Battaglia
- D. Narayanan
- Mohammad Shoeybi
- Jure Leskovec
- Alvaro Sanchez-Gonzalez
In Electrical and Electronic Engineering
- Matei A. Zaharia
- Jure Leskovec
- Sven Gowal
- Timothy A. Mann
- Dan A. Calian
- Amar Phanishayee
- Florian Stimberg
- Sylvestre-Alvise Rebuffi
In Optimization & Training
- Mohammad Shoeybi
- Bryan Catanzaro
- M. Patwary
- Matei A. Zaharia
- Alvaro Sanchez-Gonzalez
- Jure Leskovec
- Peter W. Battaglia
- Tobias Pfaff
Citations in the corpus
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021, background): “6 [28], sharded data parallelism [24, 44], where optimizer state is sharded over data-parallel workers, was introduced.”
- Learning to Simulate Complex Physics with Graph Networks (2020, background): “TPUs allowed for faster training through fast batch parallelism (each element of a batch on a separate TPU) and model parallelism (a single training distributed over multiple chips)(Kumar et al., 2019).”
- Improving Robustness using Generated Data (2021, methodology): “We use a batch size of 1024 split over 128 Google Cloud TPUv3 cores [44].”