Length bias in Encoder Decoder Models and a Case for Global Conditioning

By Pavel Sountsov, Sunita Sarawagi. 2016

43 citations recorded by the source corpus.

Encoder-decoder networks are popular for modeling sequences probabilistically in many applications. These models use the power of the Long Short-Term Memory (LSTM) architecture to capture the full dependence among variables, unlike earlier models like CRFs that typically assumed conditional independence among non-adjacent variables. However in practice encoder-decoder models exhibit a bias towards short sequences that surprisingly gets worse with increasing beam size. In this paper we show that such phenomenon is due to a discrepancy between the full sequence margin and the per-element margin enforced by the locally conditioned training objective of a encoder-decoder model. The discrepancy more adversely impacts long sequences, explaining the bias towards predicting short sequences. For the case where the predicted sequences come from a closed set, we show that a globally conditioned model alleviates the above problems of encoder-decoder models. From a practical point of view, our proposed model also eliminates the need for a beam-search during inference, which reduces to an efficient dot-product based search in a vector-space.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

AmbigQA: Answering Ambiguous Open-domain Questions (2020, background): “This problem has also been reported in other conditional generation tasks (Sountsov and Sarawagi, 2016; Stahlberg and Byrne, 2019); we leave it for future work.”
Adversarial Evaluation of Dialogue Models (2017, background): “This is relevant because it has been previously documented t hat sequence-to-sequence models have a length bias [10].”
Language Model Evaluation Beyond Perplexity (2021, background): “guage generation models (and by those in these experiments) has been claimed to result in models that favor shorter than typical sequences (Sountsov and Sarawagi, 2016; Murray and Chiang, 2018).”
On NMT Search Errors and Model Errors: Cat Got Your Tongue? (2019, methodology): “Our pre-processing is as described by Stahlberg et al. (2018a) and includes joint subword segmentation using byte pair encoding (Sennrich et al.”
If Beam Search Is the Answer, What Was the Question? (2020, background): “Other authors have ascribed fault to the model architectures, or how they are trained (Cho et al., 2014; Bengio et al., 2015; Sountsov and Sarawagi, 2016; Vinyals et al., 2017; Ott et al., 2018a; Kumar and Sarawagi, 2019).”
Better Conversations by Modeling, Filtering, and Optimizing for Coherence and Diversity (2018, background): “” in Figure 1), mainly due to the fact that neural systems tend to settle for the most frequent options, thus penalizing length and favoring high-frequency word sequences (Sountsov and Sarawagi, 2016; Wei et al., 2017).”
Neural Machine Translation: A Review (2019, background): “Sountsov and Sarawagi [74] argue that this model error is due to the locally normalized maximum likelihood training objective in NMT that underestimates the margin between the correct translation and shorter ones if trained with regularization and finite data.”
ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, background): “Language models sometimes have a contrary length bias where it favors generating shorter sequences, e.g., Sountsov and Sarawagi [2016] found encoder-decoder models tend to generate shorter sequences with beam search.”
End-to-End Speech Recognition: A Survey (2023, background): “Very small beam sizes also partly mask a length bias exhibited by E2E attention-based encoder-decoder models [279], [280], thus trading model errors against search errors [281].”
Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation (2020, methodology): “This is based on observations such as: i) translations produced via beam search typically under-estimate sequence length (Sountsov and Sarawagi, 2016; Koehn and Knowles, 2017), the length bias; ii) translation quality generally deteriorates with better approximate search (Koehn and Knowles, 2017;…”
Consistency of a Recurrent Language Model with Respect to Incomplete Decoding (2020, methodology): “Despite this success, MLE-trained neural sequence models have been shown to exhibit issues such as length bias (Sountsov & Sarawagi, 2016; Stahlberg & Byrne, 2019) and degenerate repetition (Holtz-man et al., 2019).”