An Empirical Study of Adequate Vision Span for Attention-Based Neural Machine Translation
By Raphael Shu, Hideki Nakayama. 2016
7 citations recorded by the source corpus.
Recently, the attention mechanism plays a key role to achieve high performance for Neural Machine Translation models. However, as it computes a score function for the encoder states in all positions at each decoding step, the attention model greatly increases the computational complexity. In this paper, we investigate the adequate vision span of attention models in the context of machine translation, by proposing a novel attention framework that is capable of reducing redundant score computation dynamically. The term “vision span”’ means a window of the encoder states considered by the attention model in one step. In our experiments, we found that the average window size of vision span can be reduced by over 50% with modest loss in accuracy on English-Japanese and German-English translation tasks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Armand Joulin
- Edouard Grave
- Piotr Bojanowski
- Luke S. Zettlemoyer
- Jason Aaron Edward Weston
- Douwe Kiela
- Kyunghyun Cho
- Mohit Bansal
In Computer Vision and Pattern Recognition
- Armand Joulin
- Luc Van Gool
- Xiaogang Wang
- Piotr Bojanowski
- Dacheng Tao
- Trevor Darrell
- Andrew Zisserman
- Cordelia Schmid
In Machine Translation
Citations in the corpus
- Adaptive Attention Span in Transformers (2019, methodology): “Recent progress was made with a new autoregressive model called Sequential Transformer (Vaswani et al., 2017). A Transformer is made of a sequence of layers that are composed of a block of parallel self-attention layers followed by a feedforward network. We refer to Vaswani et al. (2017) for the det”