A^3: Accelerating Attention Mechanisms in Neural Networks with Approximation
By Tae Jun Ham, Sungjun Jung, Seonghak Kim, Young H. Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W. Lee and 1 more. International Symposium on High-Performance Computer Architecture, 2020
310 citations recorded by the source corpus.
With the increasing computational demands of the neural networks, many hardware accelerators for the neural networks have been proposed. Such existing neural network accelerators often focus on popular neural network types such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs); however, not much attention has been paid to attention mechanisms, an emerging neural network primitive that enables neural networks to retrieve most relevant information from a knowledge-base, external memory, or past states. The attention mechanism is widely adopted by many state-of-the-art neural networks for computer vision, natural language processing, and machine translation, and accounts for a large portion of total execution time. We observe today's practice of implementing this mechanism using matrix-vector multiplication is suboptimal as the attention mechanism is semantically a content-based search where a large portion of computations ends up not being used. Based on this observation, we design and architect A3, which accelerates attention mechanisms in neural networks with algorithmic approximation and hardware specialization. Our proposed accelerator achieves multiple orders of magnitude improvement in energy efficiency (performance/watt) as well as substantial speedup over the state-of-the-art conventional hardware.
Subfields: General Health Professions, Philosophy
View Paper: https://www.semanticscholar.org/paper/d3c6c635b9cfd8890c7244d3db4be53d45944963
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020, methodology): “There are also hardware accelerators (Ham et al., 2020; Zhang et al., 2020) for attention and fully-connected layers in the Transformer to achieve efficient processing.”