Classical Structured Prediction Losses for Sequence to Sequence Learning

By Sergey Edunov, Myle Ott, Michael Auli, David Grangier, Marc'Aurelio Ranzato. 2017

189 citations recorded by the source corpus.

There has been much recent work on training neural attention models at the sequence-level using either reinforcement learning-style methods or by optimizing the beam. In this paper, we survey a range of classical objective functions that have been widely used to train linear models for structured prediction and apply them to neural sequence to sequence models. Our experiments show that these losses can perform surprisingly well by slightly outperforming beam search optimization in a like for like setup. We also report new state of the art results on both IWSLT’14 German-English translation as well as Gigaword abstractive summarization. On the large WMT’14 English-French task, sequence-level training achieves 41.5 BLEU which is on par with the state of the art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “FAIRSEQ has been used in many applications, such as machine translation (Gehring et al., 2017; Edunov et al., 2018b,a; Chen et al., 2018; Ott et al., 2018a; Song et al., 2018; Wu et al., 2019), language modeling (Dauphin et al., 2017; Baevski and Auli, 2019), abstractive document summarization (Fan…”
Residual Energy-Based Models for Text Generation (2020, background): “This approach is also related to other sequence level training objectives (Edunov et al., 2018), with the major differ-2 ence that in those works training aims at improving the baseline model, but generation at test time is still greedy.”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “Our self-attention baseline is the fairseq re-implementation of the Transformer Big architecture (Ott et al., 2018).”
Depth-Adaptive Transformer (2019, neutral): “We use the setup of Edunov et al. (2018) and train on 160K sentence pairs.”
Abstractive Summarization of Reddit Posts with Multi-level Memory Networks (2018, background): “We smooth the target distribution with a uniform prior distribution u (Pereyra et al., 2017; Edunov et al., 2017; Vaswani et al., 2017).”
Optimal Completion Distillation for Sequence Learning (2018, background): “In addition, alternative approaches typically do not offer a substantial performance improvement over a well tuned MLE baseline, especially when label smoothing (Pereyra et al., 2017; Edunov et al., 2018) or scheduled sampling (Bengio et al.”
Analyzing Uncertainty in Neural Machine Translation (2018, positive): “We also obtain similar results with models trained with sequence-level losses (Edunov et al., 2018).”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “Recent works have explored applying RL on tasks such as question-answering [84, 85, 47, 83, 84], summarization [58, 53, 70, 60, 17, 51], and machine translation [58, 87, 79, 82, 81, 13, 66, 5, 49].”
Random Feature Attention (2021, methodology): “(cid:28) Typically D should be no less than d in order for reasonable approximation (Yu et al., 2016); In a transformer model, d is the size of an attention head, which is usually around 64 or 128 (Vaswani et al., 2017; Ott et al., 2018).”
Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue (2020, methodology): “We apply label smoothing (Pereyra et al., 2017; Edunov et al., 2017; Vaswani et al., 2017) for both knowledge selection and response generation, and set 0.1 and 0.05 for each.”
Sequence to Sequence Mixture Model for Diverse Machine Translation (2018, methodology): “As an effective regularization strategy, we adopt label smoothing to strengthen generalisation performance (Szegedy et al., 2016; Pereyra et al., 2017; Edunov et al., 2018).”
Energy-Based Models for Text (2020, background): “This approach is also related to other sequence level training objectives (Edunov et al., 2018), with the major difference that in those works training aims at improving the baseline model, but generation at test time is still greedy.”
Zero-Shot Cross-Lingual Abstractive Sentence Summarization through Teaching Generation and Attention (2019, background): “…features and large vocabulary set (Nallapati et al., 2016), global training procedures on the sentence level (Ayana et al., 2016; Li et al., 2018; Edunov et al., 2018; Wang et al., 2018), topic enhancement in the summaries (Wang et al., 2018), additional selective gate networks in the encoder…”
Residual Energy-Based Models for Text (2021, background): “This approach is also related to other sequence-level training objectives (Edunov et al., 2018), with the major diﬀerence that those works aim at improving the baseline model, but generation at test time is still only auto-regressive.”
Empirical Analysis of Beam Search Performance Degradation in Neural Sequence Models (2019, background): “Recently, there have been several proposals for alternative models and training schemes that rely on sequence level losses and that can potentially mitigate the effects of the biases described above (Ranzato et al., 2016; Wiseman & Rush, 2016; Edunov et al., 2018).”
Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel (2019, methodology): “For the choice of datasets, we pick IWSLT’14 German-English (De-En) dataset (Edunov et al., 2017) for NMT and WikiText-103 dataset (Merity et al., 2016) for SP as suggested by Edunov et al. (Edunov et al., 2017) and Dai et al. (Dai et al., 2019).”