MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks

By Hanhwi Jang, Joonsung Kim, Jae-Eon Jo, Jaewon Lee, Jang-Hyun Kim. 2019

73 citations recorded by the source corpus.

Memory-augmented neural networks are getting more attention from many researchers as they can make an inference with the previous history stored in memory. Especially, among these memory-augmented neural networks, memory networks are known for their huge reasoning power and capability to learn from a large number of inputs rather than other networks. As the size of input datasets rapidly grows, the necessity of large-scale memory networks continuously arises. Such large-scale memory networks provide excellent reasoning power; however, the current computer infrastructure cannot achieve scalable performance due to its limited system architecture. In this paper, we propose MnnFast, a novel system architecture for large-scale memory networks to achieve fast and scalable reasoning performance. We identify the performance problems of the current architecture by conducting extensive performance bottleneck analysis. Our in-depth analysis indicates that the current architecture suffers from three major performance problems: high memory bandwidth consumption, heavy computation, and cache contention. To overcome these performance problems, we propose three novel optimizations. First, to reduce the memory bandwidth consumption, we propose a new column-based algorithm with streaming which minimizes the size of data spills and hides most of the off-chip memory accessing overhead. Second, to decrease the high computational overhead, we propose a zero-skipping optimization to bypass a large amount of output computation. Lastly, to eliminate the cache contention, we propose an embedding cache dedicated to efficiently cache the embedding matrix. Our evaluations show that MnnFast is significantly effective in various types of hardware: CPU, GPU, and FPGA. MnnFast improves the overall throughput by up to 5.38×, 4.34×, and 2.01× on CPU, GPU, and FPGA respectively. Also, compared to CPU-based MnnFast, our FPGA-based MnnFast achieves 6.54× higher energy efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Model Architectures

Citations in the corpus

Self-attention Does Not Need $O(n^2)$ Memory (2021, positive): “After publishing our initial draft, we were made aware that Jang et al. (2019) already observed that the division of the softmax operation can be delayed until the end of the attention operation (“lazy softmax”), similar to our Equation (1).”
RWKV: Reinventing RNNs for the Transformer Era (2023, methodology): “Some recent works like FlashAttention (Dao et al., 2022a) and others (Rabe and Staats, 2022; Jang et al., 2019) share similarities with RWKV’s chunked computation scheme.”
SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning (2020, background): “Previous state-of-the-art attention accelerators A(3) [6] and MNNFast [7] also leverage sparsity.”
RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing (2019, background): “MnnFast targets optimization for memory-augmented neural network and proposes a dedicated embedding cache to eliminate the cache contention between embedding and inference operations [25].”
AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators (2020, background): “…Stitch-X [76], PIM-DNN [77], Scalpel [78], CirCNN [79], CoSMIC [80], SnaPEA [81], OLAcce [82], Prediction-based DNN [83], PERMDNN [84], MnnFast [141], DNN Computation Reuse [142], vDNN [143], Compressing-DMA-Engine [144], AxTrain [145], Eager pruning [146], Bit-Tactical [147], GENESYS [148]…”