Ten Lessons From Three Generations Shaped Google’s TPUv4i : Industrial Product

By N. Jouppi, Doe Hyun Yoon, Matthew B. Ashcraft, Mark Gottscho, T. Jablin, George Kurian, James Laudon, Sheng Li, Peter C. Ma, Xiaoyu Ma and 6 more. 2021

445 citations recorded by the source corpus.

Google deployed several TPU generations since 2015, teaching us lessons that changed our views: semi-conductor technology advances unequally; compiler compatibility trumps binary compatibility, especially for VLIW domain-specific architectures (DSA); target total cost of ownership vs initial cost; support multi-tenancy; deep neural networks (DNN) grow 1.5X annually; DNN advances evolve workloads; some inference tasks require floating point; inference DSAs need air-cooling; apps limit latency, not batch size; and backwards ML compatibility helps deploy DNNs quickly. These lessons molded TPUv4i, an inference DSA deployed since 2020.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Hardware and Architecture, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

1. Sven Gowal
2. Ankur Bapna
3. Xinyun Chen
4. Bo Li
5. Pushmeet Kohli
6. Nate Kushman
7. Oriol Vinyals
8. Johan Schalkwyk

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Quoc V. Le
3. Xiaogang Wang
4. Dacheng Tao
5. Trevor Darrell
6. Ming-Hsuan Yang
7. Jitendra Malik
8. Andrew Zisserman

In Hardware and Architecture

1. Bo Li
2. Nate Kushman
3. Quoc V. Le
4. Sven Gowal
5. Ankur Bapna
6. Xinyun Chen
7. Pushmeet Kohli
8. David A. Patterson

In ML Systems & Hardware

1. Pushmeet Kohli
2. Quoc V. Le
3. Bo Li
4. Oriol Vinyals
5. David A. Patterson
6. N. Jouppi
7. Nando de Freitas
8. Tara N. Sainath

Citations in the corpus

• Competition-level code generation with AlphaCode (2022, methodology): “For each contest, we simulated running live with 3750 TPUv4 and 3750 TPUv4i (Jouppi et al., 2021) chips for the duration of the contest.”
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, background): “This result demonstrates that the USM-CTC can be used as an offline transcriber efficiently on TPUs (or GPUs).”
• MAGVIT: Masked Generative Video Transformer (2022, unknown): “When running on a single TPUv4i [36], MAGVIT-B runs at 190 fps and MAGVIT-L runs at 65 fps.”
• The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink (2022, methodology): “A DRAM access is ~6000x energy of a 16b FLOP (1300 vs 0.21 picoJoules) [10].”
• TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings (2023, methodology): “Columns 2 to 4 show workloads for inference [25], training and inference [26], and inference [27].”
• The Computational Limits of Deep Learning (2020, background): “5× in compute per dollar from 2017 to 2019[73, 74] and 4.”