Dissecting the Graphcore IPU Architecture via Microbenchmarking

By Zhe Jia, Blake Tillman, Marco Maggioni, Daniele Paolo Scarpazza. 2019

161 citations recorded by the source corpus.

This report focuses on the architecture and performance of the Intelligence Processing Unit (IPU), a novel, massively parallel platform recently introduced by Graphcore and aimed at Artificial Intelligence/Machine Learning (AI/ML) workloads. We dissect the IPU's performance behavior using microbenchmarks that we crafted for the purpose. We study the IPU's memory organization and performance. We study the latency and bandwidth that the on-chip and off-chip interconnects offer, both in point-to-point transfers and in a spectrum of collective operations, under diverse loads. We evaluate the IPU's compute power over matrix multiplication, convolution, and AI/ML primitives. We discuss actual performance in comparison with its theoretical limits. Our findings reveal how the IPU's architectural design affects its performance. Moreover, they offer simple mental models to predict an application's performance on the IPU, on the basis of the computation and communication steps it involves. This report is the natural extension to a novel architecture of a continuing effort of ours that focuses on the microbenchmark-based discovery of massively parallel architectures.

Subfields: Electrical and Electronic Engineering, Hardware and Architecture, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Hardware and Architecture

In ML Systems & Hardware

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “Performance on other hardware accelerators are similar [48, 50].”
Unit Scaling: Out-of-the-Box Low-Precision Training (2023, methodology): “Hardware & distributed training Models were trained on IPU hardware (Jia et al., 2019), using either Bow Pod16 or IPU-POD16 Classic machines.”
Pipelined Backpropagation at Scale: Training Large Models without Batches (2020, background): “Architectures with distributed low latency memory architectures, such as (Lie, 2020; Vassilieva, 2020; Sambanova, 2020; Nicol, 2017; Jia et al., 2019), could benefit from pipelined training approaches.”
The Deep Learning Compiler: A Comprehensive Survey (2020, methodology): “erators also maintain their specific kernel libraries [22], [23].”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Tensor processing units (TPU) from Google [119], Habana form Intel [159], Kunlun from Baidu [181], Hanguang from Alibaba [118], and intelligence processing unit (IPU) from Graphcore [115].”
Data Movement Is All You Need: A Case Study on Optimizing Transformers (2020, methodology): “Beyond these, TASO (Jia et al., 2019a) targets similar optimizations to ours by using graph substitutions, but considers only inference and does not exhaustively explore the search space.”