A domain-specific supercomputer for training deep neural networks

By N. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, David A. Patterson. 2020

294 citations recorded by the source corpus.

Google's TPU supercomputers train deep neural networks 50x faster than general-purpose supercomputers running a high-performance computing benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In ML Systems & Hardware

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Hardware & Software: Training Hardware: TPU v4 (Jouppi et al., 2020).”
Pathways: Asynchronous Distributed Dataflow for ML (2022, methodology): “We construct a more realistic pipeline benchmark in which the simple computations from the earlier benchmark are again chained together, but now each computation runs on a different set of 4 TPU cores, each on a different host, and data output from one computation must be sent via ICI before the nex”
Scaling Instruction-Finetuned Language Models (2022, background): “Implementation Frameworks Hardware & Software for Training Hardware: TPU v3 or TPU v4 (Jouppi et al., 2020).”
Mixture-of-Experts with Expert Choice Routing (2022, methodology): “The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19].”
UL2: Unifying Language Learning Paradigms (2022, methodology): “Each pretraining run is typically trained using 64 to 128 TPUv4 chips (Jouppi et al., 2020).”
PaLM 2 Technical Report (2023, neutral): “Implementation Frameworks Hardware & Software Hardware: TPU v4 (Jouppi et al., 2020).”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, neutral): “Implementation Frameworks Hardware & Software Hardware: TPU v4 (Jouppi et al., 2020).”
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text (2022, methodology): “The models are trained on 64 Cloud v4 TPUs (Jouppi et al., 2020).”
Physics-informed machine learning: case studies for weather and climate modelling (2021, background): “Computationally, how do we scale the training, testing and deployment of complex PIML models on large datasets efficiently, so that they perform well in a rapidly changing computational landscape [130]? Resource-wise, how can we effectively collaborate across many diverse communities: physicists, ma”
TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings (2023, background): “Since prior papers have described the fundamentals of previous TPUs for training [26, 39] and for inference [25, 27], this paper focuses on the three novel features of TPU v4 listed above”
Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models (2022, background): “dedicated InterChip Interconnects (ICI) links on TPUs that provide high bandwidth on both directions [26, 36], which helps improving the system performance with the proposed overlapping technique in general.”
The Computational Limits of Deep Learning (2020, background): “5× in compute per dollar from 2017 to 2019[73, 74] and 4.”
Muse: Text-To-Image Generation via Masked Generative Transformers (2023, methodology): “Training is performed for 1M steps, with a batch size of 512 on 512-core TPU-v4 chips (Jouppi et al., 2020).”
Injecting Text in Self-Supervised Speech Pretraining (2021, methodology): “Pretraining Parameters: All ASR models are trained on Google TPU V3 cores [54].”
Searching for Fast Model Families on Datacenter Accelerators (2021, methodology): “With the improved NAS and LACS, we search for high accuracy CNNs for efficient inference on TPUv3 [20, 15, 42] and GPUv100 [14].”
Genie: Generative Interactive Environments (2024, methodology): “For this experiment we make use of TPUv2 and TPUv3 (Jouppi et al., 2020).”
Scaling Vision Transformers to 22 Billion Parameters (2023, background): “Hardware & Software: Deployment Hardware: TPU v4 (Jouppi et al., 2020).”
Ten Lessons From Three Generations Shaped Google’s TPUv4i : Industrial Product (2021, background): “Training needs large scale, so another enhancement is to add a custom chip-to-chip interconnect fabric ( ICI ), enabling TPUv2 supercomputers of up to 256 chips [23].”
Unifying Language Learning Paradigms (2022, methodology): “Each pre-training run is typically trained using 64 to 128 TPUv4 chips (Jouppi et al., 2020).”