Twin Networks: Matching the Future for Sequence Generation

By Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, C. Pal, Yoshua Bengio. 2017

69 citations recorded by the source corpus.

Subfields: Natural Language Processing, Model Architectures, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Dong Yu
2. Yu Zhang
3. Jianfeng Gao
4. Graham Neubig
5. Zhe Gan
6. Tara N. Sainath
7. Jinyu Li
8. Mohit Bansal

In Model Architectures

1. Dong Yu
2. Tara N. Sainath
3. Yu Zhang
4. William Chan
5. Yonghui Wu
6. Quoc V. Le
7. Yoshua Bengio
8. Zhe Gan

In Natural Language Generation

1. Mohit Iyyer
2. Dong Yu
3. Chung-Cheng Chiu
4. John Wieting
5. Nan Duan
6. Yonghui Wu
7. Marzena Karpinska
8. Machel Reid

Citations in the corpus

• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, methodology): “Methods for improving the performance of streaming ASR models using models with future context have been studied in [44]–[52].”
• Knowledge Distillation from Offline to Streaming RNN Transducer for End-to-End Speech Recognition (2020, methodology): “For attention-based encoder-decoder and hybrid RNN/HMM, twin regularization to make forward hidden states as close as backward hidden states has been applied [30,31].”
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training (2020, methodology): “However, as discussed in previous works (Pascanu et al., 2013; Gulcehre et al., 2017; Serdyuk et al., 2018), AR-based models may prefer to focus on the latest tokens rather than capture long-term dependencies for the next token prediction.”
• RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “…methods that predict multiple to-kens (Oord et al., 2018; Qi et al., 2020), rely on retrieval (Khandelwal et al., 2020), use bidirectional LMs (Serdyuk et al., 2018), employ contrastive learning (Su et al., 2022; An et al., 2022), use BERT for sentence-level language modeling (Ip-polito et…”
• Modeling Fluency and Faithfulness for Diverse Neural Machine Translation (2019, methodology): “ecode forwards and backwards simultaneously. The main difference from our work is that they require decoding decoding bi-directionally while our method can decode once just like Transformer. Although Serdyuk et al. (2018) also employ bidirectional decoding during training, they instead use the backw”
• Distilling the Knowledge of BERT for Text Generation (2019, methodology): “Twin Networks (Serdyuk et al., 2018) used a backward RNN jointly trained with a forward RNN decoder by matching their hidden states.”
• Distilling Knowledge Learned in BERT for Text Generation (2019, methodology): “Twin Networks (Serdyuk et al., 2018) used a backward RNN jointly trained with a forward RNN decoder by matching their hidden states.”
• Fraternal Dropout (2017, methodology): “Another more recent way of regularizing RNNs, that is similar in spirit to the approach we take, involves minimizing the difference between the hidden states of the original and the auxiliary network Serdyuk et al. (2017).”