Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks

By Yu-hsin Chen, Joel S. Emer, Vivienne Sze. 2016

1,634 citations recorded by the source corpus.

Deep convolutional neural networks (CNNs) are widely used in modern AI systems for their superior accuracy but at the cost of high computational complexity. The complexity comes from the need to simultaneously process hundreds of filters and channels in the high-dimensional convolutions, which involve a significant amount of data movement. Although highly-parallel compute paradigms, such as SIMD/SIMT, effectively address the computation requirement to achieve high throughput, energy consumption still remains high as data movement can be more expensive than computation. Accordingly, finding a dataflow that supports parallel processing with minimal data movement cost is crucial to achieving energy-efficient CNN processing without compromising accuracy. In this paper, we present a novel dataflow, called row-stationary (RS), that minimizes data movement energy consumption on a spatial architecture. This is realized by exploiting local data reuse of filter weights and feature map pixels, i.e., activations, in the high-dimensional convolutions, and minimizing data movement of partial sum accumulations. Unlike dataflows used in existing designs, which only reduce certain types of data movement, the proposed RS dataflow can adapt to different CNN shape configurations and reduces all types of data movement through maximally utilizing the processing engine (PE) local storage, direct inter-PE communication and spatial parallelism. To evaluate the energy efficiency of the different dataflows, we propose an analysis framework that compares energy cost under the same hardware area and processing parallelism constraints. Experiments using the CNN configurations of AlexNet show that the proposed RS dataflow is more energy efficient than existing dataflows in both convolutional (1.4× to 2.5×) and fully-connected layers (at least 1.3× for batch size larger than 16). The RS dataflow has also been demonstrated on a fabricated chip, which verifies our energy analysis.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Efficiency & Compression

Citations in the corpus

In-datacenter performance analysis of a tensor processing unit (2017, background): “The more recent DianNao family of NN architectures minimizes memory accesses both on the chip and to external DRAM by having efficient architectural support for the memory access patterns that appear in NN applications [28, 11].”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “…Reconﬁgurable Systems, and ASICs for AI: There is a wide variety of system hardware choices for AI from general-purpose processors (CPUs), general-purpose accelerators (GPUs or TPUs), ﬁeld-programmable gate arrays (FPGAs) [81], to application-speciﬁc integrated circuit (ASIC), such as Eyeriss [82].”
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications (2018, methodology): “However, this requires designing models to estimate the direct metrics of each target platform, which requires detailed knowledge of the platform including its hardware architecture [3], and the network-to-array mapping used in the toolchain [2].”
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning (2016, methodology): “The total number of memory accesses is a function of the CNN shape configuration [21] (i.”
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning (2018, background): “Learning Accelerator (VDLA) – which distills characteristics from previous accelerator proposals [12, 21, 27] into a minimalist hardware architecture – to demonstrate TVM’s ability to generate highly efficient schedules that can target specialized accelerators.”
An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection (2019, neutral): “[1], the memory footprint is attributed to both filter parameter and intermediate feature maps.”
Scalpel: Customizing DNN pruning to the underlying hardware parallelism (2017, neutral): “For hardware-based methods, different ASIC designs [2, 5, 7, 8, 10, 14, 18, 25, 27, 33, 36, 37] and tools for accelerating DNN on existing platform have been proposed.”
End-to-End Learning of Energy-Constrained Deep Neural Networks (2018, methodology): “Many DNN hardware architectures adopt the systolic array, most notably the Google Tensor Processing Unit (TPU) [19].”
Machine Learning at Facebook: Understanding Inference at the Edge (2019, background): “Recent work on hardware for machine learning and efficient training and inference has substantially advanced the state-of-the-art [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66].”
Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions (2018, methodology): “Therefore, we ﬁrst normalize the representational cost in terms of the computational cost assuming that a global on-chip buffer is employed, implying that the representational cost of a MAC is about 6 times that of performing a MAC computation (Chen et al., 2016b).”
AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators (2020, background): “Some designs investigate the dataﬂow (or data reuse pattern) in DNN workloads [51–59], among which Eyeriss [52–54] is a representative design that explores many data reuse opportunities existed in DNN execution.”
Gist: Efficient Data Encoding for Deep Neural Network Training (2018, methodology): “Many HW accelerators have been designed employing limited precision and leveraging sparsity to reduce computational and memory requirements [17], [5], [6], [35], [29], [38], [47], [2].”
SnaPEA: Predictive Early Activation for Reducing Computation in Deep Convolutional Neural Networks (2018, methodology): “For instance, assigning a label to a relatively small RGB image (224 × 224 × 3) from the ImageNet database [1] requires billions of multiply-and-accumulate operations [2]–[4].”
Binary Neural Networks: A Survey (2020, background): “Although at present, a large amount of dedicated hardware emerges for deep learning [16, 17, 18, 19, 20], providing eﬃcient vector operations to enable fast convolution in forward inference, the heavy computation and storage still inevitably limit the applications of the deep CNNs in practice.”
Exploring Energy and Accuracy Tradeoff in Structure Simplification of Trained Deep Neural Networks (2018, neutral): “In [2], Eyeriss architecture is proposed which minimizes energy consumption associated with data movement.”
Socio-Technological Challenges and Opportunities: Paths Forward (2021, neutral): “…of AI have revolutionized the entire system stack design, resulting in hardware tailored to machine learning execution [Ovtcharov et al., 2015, Chen et al., 2016, Jouppi et al., 2017, Fowers et al., 2018, Shao et al., 2019, Mattson et al., 2020, Henry et al., 2020, Reddi et al., 2021,…”
Escher: A CNN Accelerator with Flexible Buffering to Minimize Off-Chip Transfer (2017, methodology): “[10], [23] use NoCs with FMA compute nodes.”
TETRIS: Scalable and Efficient Neural Network Acceleration with 3D Memory (2017, methodology): “The PEs are organized in a 2D array, with a custom onchip interconnect between the PEs and the memory hierarchy [5, 7, 12].”