A Configurable Cloud-Scale DNN Processor for Real-Time AI

By J. Fowers, Kalin Ovtcharov, Michael Papamichael, Todd Massengill, Ming Liu, Daniel Lo, Shlomi Alkalay, M. Haselman, Logan Adams, Mahdi Ghandi and 10 more. International Symposium on Computer Architecture, 2018

557 citations recorded by the source corpus.

Interactive AI-powered services require low-latency evaluation of deep neural network (DNN) models—aka ""real-time AI"". The growing demand for computationally expensive, state-of-the-art DNNs, coupled with diminishing performance gains of general-purpose architectures, has fueled an explosion of specialized Neural Processing Units (NPUs). NPUs for interactive services should satisfy two requirements: (1) execution of DNN models with low latency, high throughput, and high efficiency, and (2) flexibility to accommodate evolving state-of-the-art models (e.g., RNNs, CNNs, MLPs) without costly silicon updates. This paper describes the NPU architecture for Project Brainwave, a production-scale system for real-time AI. The Brainwave NPU achieves more than an order of magnitude improvement in latency and throughput over state-of-the-art GPUs on large RNNs at a batch size of 1. The NPU attains this performance using a single-threaded SIMD ISA paired with a distributed microarchitecture capable of dispatching over 7M operations from a single instruction. The spatially distributed microarchitecture, scaled up to 96,000 multiply-accumulate units, is supported by hierarchical instruction decoders and schedulers coupled with thousands of independently addressable high-bandwidth on-chip memories, and can transparently exploit many levels of fine-grain SIMD parallelism. When targeting an FPGA, microarchitectural parameters such as native datapaths and numerical precision can be "synthesis specialized" to models at compile time, enabling atypically high FPGA performance competitive with hardened NPUs. When running on an Intel Stratix 10 280 FPGA, the Brainwave NPU achieves performance ranging from ten to over thirty-five teraflops, with no batching, on large, memory-intensive RNNs.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Hardware and Architecture

View Paper: https://www.semanticscholar.org/paper/1324c98f35256b3fe64fae9f777e2f7d76f194c0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Hardware and Architecture

Citations in the corpus

MASR: A Modular Accelerator for Sparse RNNs (2019, methodology): “Beyond accelerators for DNNs and CNNs, other work has investigated RNNs, and search algorithms for ASR and machine translation [31], [35], [36], [37], [38], [39], [40].”
Simba: Scaling Deep-Learning Inference with Multi-Chip-Module-Based Architecture (2019, background): “towards even larger and deeper networks, the ensuing compute and storage requirements motivate large-scale compute capability in DL hardware, which is currently addressed by a combination of large monolithic chips and homogeneous multi-chip board designs [14, 17, 24, 29, 39, 71].”
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications (2018, methodology): “In order to tackle this trend, a lot of research has been done on optimizing computing platforms for DL, including but not limited to [1, 18, 25, 33, 49, 50, 61, 62].”
Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration (2019, background): “Most existing hardware DNN architectures are largely spatial, where parallel execution units are laid out spatially either in a systolic fashion, as in the case of the TPU, or in parallel vector units like Brainwave [17] and NVDLA [11].”
MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks (2019, background): “Variants of existing DNNs usually have different requirements including new operations, which can motivate us to develop new optimized DNN accelerators [11, 15, 22, 39, 43, 64, 83].”
SIGMA: A Sparse and Irregular GEMM Accelerator with Flexible Interconnects for DNN Training (2020, methodology): “A 1D substrate enables us to run matrixmatrix (M*M) multiplications as multiple vector matrix multiplications (V*M), similar to Brainwave [14].”
Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads (2020, background): “These accelerators have been deployed to great effect [25, 35] to train state-of-the-art deep neural network (DNN) models for many domains, including language, image and video [14, 30, 31, 51, 55].”
Planaria: Dynamic Architecture Fission for Spatial Multi-Tenant Acceleration of Deep Neural Networks (2020, methodology): “In fact, INFerence-as-a-Service (INFaaS) [14], has become the backbone of the deployed applications in Voice Assistants [15, 16], Smart Speakers [17], and enterprise applications [18–20], etc. Cloud-backed inference currently dominates the market [21–24] and is enabled by various forms of custom acc”
A^3: Accelerating Attention Mechanisms in Neural Networks with Approximation (2020, background): “CNN accelerators for custom ASIC or FPGA device [1], [2], [3], [61], [62], [63], [64], [65], [66], [67] and RNN accelerators [3], [68], [69], [70] are proposed to accelerate NN processing by exploiting massive parallelism and optimizing data movements.”
Socio-Technological Challenges and Opportunities: Paths Forward (2021, methodology): “…system stack design, resulting in hardware tailored to machine learning execution [Ovtcharov et al., 2015, Chen et al., 2016, Jouppi et al., 2017, Fowers et al., 2018, Shao et al., 2019, Mattson et al., 2020, Henry et al., 2020, Reddi et al., 2021, Anderson et al., 2021, Jang et al., 2021,…”
Hardware/Software Co-Exploration of Neural Architectures (2019, background): “energy efficiency, in particular for low-batch inferences [19], [20].”
A Survey on Green 6G Network: Architecture and Technologies (2019, background): “High-performance hardware is another driving factor for RTE and a specialized real-time AI processor has been designed in [40].”
TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models (2021, background): “…there is considerable interest to design domainspecific accelerators and at-scale infrastructures (Alibaba, 2019; Amazon, 2019; Chung et al., 2018; Fowers et al., 2018; Hazelwood et al., 2018; Jouppi et al., 2017; Lee & Rao, 2019; Mattson et al., 2019; NVIDIA, 2020a; Ovtcharov et al., 2015; Reddi…”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “The neural processing unit (NPU) is presented in [329] for the project Brainwave to provide real-time AI.”
Accuracy vs. Efficiency: Achieving Both through FPGA-Implementation Aware Neural Architecture Search (2019, background): “CIFAR-10 45,000 5,000 25 10 [1,3,5,7] [24,36,48,64] 60 [1.”