Modeling Recurrence for Transformer

By Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, Zhaopeng Tu. 2019

87 citations recorded by the source corpus.

Recently, the Transformer model that is based solely on attention mechanisms, has advanced the state-of-the-art on various machine translation tasks. However, recent studies reveal that the lack of recurrence modeling hinders its further improvement of translation capacity. In response to this problem, we propose to directly model recurrence for Transformer with an additional recurrence encoder. In addition to the standard recurrent neural network, we introduce a novel attentive recurrent network to leverage the strengths of both attention models and recurrent networks. Experimental results on the widely-used WMT14 English⇒German and WMT17 Chinese⇒English translation tasks demonstrate the effectiveness of the proposed approach. Our studies also reveal that the proposed model benefits from a short-cut that bridges the source and target sequences with a single recurrent layer, which outperforms its deep counterpart.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Noah A. Smith
2. Mohit Bansal
3. Luke S. Zettlemoyer
4. Yoav Goldberg
5. Yejin Choi
6. Jianfeng Gao
7. Graham Neubig
8. Douwe Kiela

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Luc Van Gool
3. Mohit Bansal
4. Dacheng Tao
5. Cordelia Schmid
6. Andrew Zisserman
7. Trevor Darrell
8. Jitendra Malik

In Model Architectures

1. Noah A. Smith
2. Dani Yogatama
3. Hao Peng
4. Mostafa Dehghani
5. Mingda Chen
6. Samira Abnar
7. Zhenzhong Lan
8. Nikolaos Pappas

Citations in the corpus

• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “Hao et al. (2019) combine a parameter-sharing transformer with the standard one, which further increases the number of parameters of the standard transformer.”
• Random Feature Attention (2021, background): “Answering to this, many approaches have been proposed, e.g., learning the attention spans (Sukhbaatar et al., 2019; Wu et al., 2020), and enhancing the attention computation with recurrent (Hao et al., 2019; Chen et al., 2019) or convolutional (Wu et al., 2019; Mohamed et al., 2019) components.”
• Attention mechanism in neural networks: where it comes and where it goes (2022, methodology): “For instance, an additional recurrence encoder is preferred to model recurrence for Transformer directly [113].”
• Transferring Inductive Biases through Knowledge Distillation (2020, methodology): “The recursion in a model can be implemented in different ways, like in Recurrent Neural Networks [9], Recursive Neural Networks [34, 22] and Universal Transformers [6, 15].”
• Theoretical Limitations of Self-Attention in Neural Sequence Models (2019, methodology): “The idea of using such input restrictions has been successful in the theory of Boolean circuits (Furst et al., 1984; Hastad et al., 1994).”
• Transformers in Time-Series Analysis: A Tutorial (2022, background): “However, the intuition behind the basic operations used remains the same and is covered in this Section [39, 40, 41].”
• Neural Machine Translation: A Review (2019, methodology): “[134] who used a Transformer encoder and a recurrent encoder in parallel, and Lin et al.”