Dense Information Flow for Neural Machine Translation

By Yanyao Shen, Xu Tan, Di He, Tao Qin, Tie-Yan Liu. 2018

36 citations recorded by the source corpus.

Recently, neural machine translation has achieved remarkable progress by introducing well-designed deep neural networks into its encoder-decoder framework. From the optimization perspective, residual connections are adopted to improve learning performance for both encoder and decoder in most of these deep architectures, and advanced attention connections are applied as well. Inspired by the success of the DenseNet model in computer vision problems, in this paper, we propose a densely connected NMT architecture (DenseNMT) that is able to train more efficiently for NMT. The proposed DenseNMT not only allows dense connection in creating new features for both encoder and decoder, but also uses the dense attention structure to improve attention quality. Our experiments on multiple datasets show that DenseNMT structure is more competitive and efficient.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Yi Ren
5. Zhou Zhao
6. Yingce Xia
7. Lijun Wu
8. Di He

In Computer Vision and Pattern Recognition

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Lijun Wu
5. Yingce Xia
6. Fei Tian
7. Di He
8. J. Lai

In Natural Language Processing

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Lijun Wu
5. Yingce Xia
6. Di He
7. Fei Tian
8. J. Lai

Citations in the corpus

• MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, background): “, 2017) is a challenging task in artificial intelligence, and covers a variety of language generation applications such as NMT (Cho et al., 2014; Bahdanau et al., 2015a; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Tan et al., 2019; Artetxe et al., 2017; Lample et al., 2017; 2018; He”
• Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “Neural Machine Translation (NMT) has witnessed rapid development in recent years (Bahdanau et al., 2015; Luong et al., 2015b; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Wu et al., 2018; Song et al., 2018; Shen et al., 2018; Guo et al., 2018; He et al., 2018; Gong et al., 2018), inc”
• Multilingual Neural Machine Translation with Language Clustering (2019, methodology): “The encoder-decoder framework (Bahdanau et al., 2015; Luong et al., 2015b; Sutskever et al., 2014; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Shen et al., 2018; He et al., 2018) is adopted to model the conditional probability P (y|x, θ), where the encoder maps the input to a set of”
• A Study of Multilingual Neural Machine Translation (2019, background): “…recent years (Bahdanau, Cho, and Bengio 2015; Luong, Pham, and Manning 2015; Sutskever, Vinyals, and Le 2014; Wu et al. 2016; Gehring et al. 2017; Shen et al. 2018; Vaswani et al. 2017; He et al. 2018; Hassan et al. 2018) for single-pair translation with a large number of bilingual sentence…”
• Efficient Bidirectional Neural Machine Translation (2019, background): “Neural Machine Translation (NMT) is very challenging and attracts lots of research attention in recent years (Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Gehring et al., 2017a; Vaswani et al., 2017a; Shen et al., 2018; He et al., 2018; Hassan et al., 2018; Song et al., 2019).”
• Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation (2018, background): “Neural Machine Translation (NMT) is a challenging task that attracts a lot of attention in recent years [5, 2, 18, 29, 7, 33, 28, 27, 35, 37, 9], and the structure of NMT models has evolved quickly.”
• Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter (2018, background): “Neural machine translation (NMT) has attracted much research attention in recent years (Bahdanau et al., 2014; Shen et al., 2018; Song et al., 2018; Xia et al., 2018; He et al., 2016; Wu et al., 2017, 2018).”
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input (2018, background): “…models have an inherently sequential architecture that prevents them from being parallelized during training and inference, which is partially solved by CNNs and self-attention based models (Kalchbrenner et al. 2016; Gehring et al. 2017; Shen et al. 2018; Vaswani et al. 2017; Song et al. 2018).”