SIGMA: A Sparse and Irregular GEMM Accelerator with Flexible Interconnects for DNN Training

By Eric Qin, A. Samajdar, Hyoukjun Kwon, V. Nadella, Sudarshan M. Srinivasan, Dipankar Das, Bharat Kaul, Tushar Krishna. International Symposium on High-Performance Computer Architecture, 2020

515 citations recorded by the source corpus.

The advent of Deep Learning (DL) has radically transformed the computing industry across the entire spectrum from algorithms to circuits. As myriad application domains embrace DL, it has become synonymous with a genre of workloads across vision, speech, language, recommendations, robotics, and games. The key compute kernel within most DL workloads is general matrix-matrix multiplications (GEMMs), which appears frequently during both the forward pass (inference and training) and backward pass (training). GEMMs are a natural choice for hardware acceleration to speed up training, and have led to 2D systolic architectures like NVIDIA tensor cores and Google Tensor Processing Unit (TPU). Unfortunately, emerging GEMMs in DL are highly irregular and sparse, which lead to poor data mappings on systolic architectures. This paper proposes SIGMA, a flexible and scalable architecture that offers high utilization of all its processing elements (PEs) regardless of kernel shape and sparsity. Within SIGMA includes a novel reduction tree microarchitecture named Forwarding Adder Network (FAN). SIGMA performs 5.7x better than systolic array architectures for irregular sparse matrices, and roughly 3x better than state-of-the-art sparse accelerators. We demonstrate an instance of SIGMA operating at 10.8 TFLOPS efficiency across arbitrary levels of sparsity, with a 65.10 mm^2 and 22.33 W footprint on a 28 nm process.

Subfields: Computer Vision and Pattern Recognition, Hardware and Architecture

View Paper: https://www.semanticscholar.org/paper/fcf1b4473a0af1f3ebc0fd556ee30c9309ff6345

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Hardware and Architecture

Citations in the corpus

AWB-GCN: A Graph Convolutional Network Accelerator with Runtime Workload Rebalancing (2019, methodology): “SIGMA [55] and ALRESCHA [56] are recent high-performance architectures for SpMM and SpMV.”
Planaria: Dynamic Architecture Fission for Spatial Multi-Tenant Acceleration of Deep Neural Networks (2020, unknown): “MAERI [52] and SIGMA [95] propose a reconfigurable interconnect among the PEs to deal with sparsity in neural networks [52] and matrix multiplications [95] and to”
Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration (2019, background): “Based on these architecture templates, recent advances have also started exploring how to leverage application’s sparsity patterns [18]–[20] and/or emerging in-memory computing technology [21], [22].”