Simple and Effective Noisy Channel Modeling for Neural Machine Translation

By Kyra Yee, Nathan Ng, Yann Dauphin, Michael Auli. 2019

83 citations recorded by the source corpus.

Previous work on neural noisy channel modeling relied on latent variable models that incrementally process the source and target sentence. This makes decoding decisions based on partial source prefixes even though the full source is available. We pursue an alternative approach based on standard sequence to sequence models which utilize the entire source. These models perform remarkably well as channel models, even though they have neither been trained on, nor designed to factor over incomplete target sentences. Experiments with neural language models trained on billions of words show that noisy channel models can outperform a direct model by up to 3.2 BLEU on WMT’17 German-English translation. We evaluate on four language-pairs and our channel models consistently outperform strong alternatives such right-to-left reranking models and ensembles of direct models.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Noisy Channel Language Model Prompting for Few-Shot Text Classification (2021, methodology): “…channel models for prompted few-shot text classiﬁcation with large language models, inspired by noisy channel models in machine translation (Brown et al., 1993; Koehn et al., 2003; Yu et al., 2017; Yee et al., 2019) and their extensions to other tasks (Yogatama et al., 2017; Lewis and Fan, 2018).”
Putting Machine Translation in Context with the Noisy Channel Model (2019, methodology): “The beneﬁts of strong language models estimated from nonparallel data has long been recognized in translation (Brants et al., 2007), and more recently in noisy channel approaches to translation based on neural network component distributions (Yu et al., 2017; Yee et al., 2019; Ng et al., 2019).”
Leveraging Large Pretrained Models for WebNLG 2020 (2020, methodology): “As shown in Shen et al. (2019); Yee et al. (2019), techniques similar to the reverse model reranking method we use here, under the guise of RSA or noisy channel models, have shown signiﬁcant beneﬁts for generation tasks when there is room for improvement.”
Building Adaptive Acceptability Classifiers for Neural NLG (2021, methodology): “As an alternative to using acceptability classi-ﬁers, one can make use of reconstruction models (Shen et al., 2019; Yee et al., 2019) to determine how well the NLG model’s output predicts its input.”
Better Document-Level Machine Translation with Bayes’ Rule (2019, methodology): “This sentreranker setup is close to the work from Yee et al. (2019) and Ng et al. (2019) with the difference that rather than using a language model trained on documents we use a language model trained on sentences, which is more statistically consistent.”
Data-to-text Generation with Macro Planning (2021, methodology): “We could alleviate this problem by adopting a noisy channel decomposition (Yee et al., 2019; Yu et al., 2020), i.e., by learning two different distributions: a conditional model which provides the probability of translating a paragraph plan to text and a language model which provides an…”
Facebook AI’s WAT19 Myanmar-English Translation Task Submission (2019, methodology): “We also report further improvements by swap-ping beam search decoding with noisy channel re-ranking (Yee et al., 2019) and by ensembling.”
Facebook AI’s WMT20 News Translation Task Submission (2020, methodology): “To rerank our system output, we use the noisy-channel model (Yee et al., 2019) as the scoring model (Ng et al., 2019; Chen et al., 2019).”
Domain Robustness in Neural Machine Translation (2019, methodology): “, 2019), subword regularization (Kudo, 2018), neural noisy channel models (Li and Jurafsky, 2016; Yee et al., 2019), and defensive distillation (Papernot et al.”
Multimodal Conditionality for Natural Language Generation (2021, background): “Models conditioned with pretrained language include noisy channel modeling (Yee et al., 2019) and fusion approaches (Sriram et al., 2017; Gulcehre et al., 2015) that concatenate hidden states of the conditional model with that of language model to predict the next word.”