Simba: Scaling Deep-Learning Inference with Multi-Chip-Module-Based Architecture

By Yakun Sophia Shao, Jason Clemons, Rangharajan Venkatesan, B. Zimmer, Matthew R. Fojtik, Nan Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Ross Pinckney, Priyanka Raina and 7 more. Micro, 2019

495 citations recorded by the source corpus.

Package-level integration using multi-chip-modules (MCMs) is a promising approach for building large-scale systems. Compared to a large monolithic die, an MCM combines many smaller chiplets into a larger system, substantially reducing fabrication and design costs. Current MCMs typically only contain a handful of coarse-grained large chiplets due to the high area, performance, and energy overheads associated with inter-chiplet communication. This work investigates and quantifies the costs and benefits of using MCMs with fine-grained chiplets for deep learning inference, an application area with large compute and on-chip storage requirements. To evaluate the approach, we architected, implemented, fabricated, and tested Simba, a 36-chiplet prototype MCM system for deep-learning inference. Each chiplet achieves 4 TOPS peak performance, and the 36-chiplet MCM package achieves up to 128 TOPS and up to 6.1 TOPS/W. The MCM is configurable to support a flexible mapping of DNN layers to the distributed compute and storage units. To mitigate inter-chiplet communication overheads, we introduce three tiling optimizations that improve data locality. These optimizations achieve up to 16% speedup compared to the baseline layer mapping. Our evaluation shows that Simba can process 1988 images/s running ResNet-50 with batch size of one, delivering inference latency of 0.50 ms.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/8efd76fafff43a1b420f8cda092a8bab8d545732

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Hai Helen Li
2. Yiran Chen
3. Tushar Krishna
4. Yuan Xie
5. Song Han
6. Guoqi Li
7. Luping P. Shi
8. By Lei Deng

In Electrical and Electronic Engineering

1. Tushar Krishna
2. Guoqi Li
3. Hyoukjun Kwon
4. Song Han
5. Nam Sung Kim
6. Yuan Xie
7. By Lei Deng
8. Hadi Esmaeilzadeh

Citations in the corpus

• Planaria: Dynamic Architecture Fission for Spatial Multi-Tenant Acceleration of Deep Neural Networks (2020, background): “Simba [96] proposes a scalable multi-chip module-based accelerator to reduce fabrication cost and provide scalability with respect to inter-chip and intra-chip communication.”
• SIGMA: A Sparse and Irregular GEMM Accelerator with Flexible Interconnects for DNN Training (2020, methodology): “The NoC connecting the Flex-DPEs together is similar to that of tiled accelerator architectures, such as Tangram [16] and Simba [39].”
• AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators (2020, background): “Due to the high demand of computing power in DL applications, we have recently witnessed a phenomenal trend in which the landscape of computing has shifted from general-purpose processors to domain-specific architectures [16–110].”
• Socio-Technological Challenges and Opportunities: Paths Forward (2021, unknown): “…resulting in hardware tailored to machine learning execution [Ovtcharov et al., 2015, Chen et al., 2016, Jouppi et al., 2017, Fowers et al., 2018, Shao et al., 2019, Mattson et al., 2020, Henry et al., 2020, Reddi et al., 2021, Anderson et al., 2021, Jang et al., 2021, Thompto et al., 2021,…”
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “The complete architecture is similar to the spatial architectures in [264]–[267] (see Fig.”