The Neural Noisy Channel

By Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, Tomás Kociský. 2016

70 citations recorded by the source corpus.

We formulate sequence to sequence transduction as a noisy channel decoding problem and use recurrent neural networks to parameterise the source and channel models. Unlike direct models which can suffer from explaining-away effects during training, noisy channel models must produce outputs that explain their inputs, and their component models can be trained with not only paired training samples but also unpaired samples from the marginal output distribution. Using a latent variable to control how much of the conditioning sequence the channel model needs to read in order to generate a subsequent symbol, we obtain a tractable and effective beam search decoder. Experimental results on abstractive sentence summarisation, morphological inflection, and machine translation show that noisy channel models outperform direct models, and that they significantly benefit from increased amounts of unpaired output data that direct models cannot easily use.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, background): “This is in part due to the recency bias and explanation-away problem, where language models have the tendency to rely overly on the short-term context that can easily explain away the longer-term context (Yu et al., 2017b).”
How Decoding Strategies Affect the Verifiability of Generated Text (2019, background): “Findings of the Association for Computational Linguistics: EMNLP 2020, pages 223–235 November 16 - 20, 2020. c©2020 Association for Computational Linguistics 223”
Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, background): “In earlier works Gu et al. (2016; 2017); Chen et al. (2018) explored the idea of using a small neural network to steer an LM. Yu et al. (2016), and more recently Yu et al. (2019); Yee et al. (2019); Ng et al. (2019), leveraged the Shannon Noisy Channel Theory (Shannon, 1948) for improving…”
Learning to Write with Cooperative Discriminators (2018, methodology): “Yu et al. (2017c) proposed a model where the speaker consists of a convolutional encoder and an LSTM decoder, trained with a ranking loss on negative samples in addition to optimizing log-likelihood.”
Online and Linear-Time Attention by Enforcing Monotonic Alignments (2017, methodology): “More closely to our model, (Yu et al., 2016b) similarly add “shift” and “emit” operations to an RNN. Finally, the Segmental RNN (Kong et al., 2015) treats a segmentation of the input sequence as a latent random variable.”
Noisy Channel Language Model Prompting for Few-Shot Text Classification (2021, methodology): “…channel models for prompted few-shot text classiﬁcation with large language models, inspired by noisy channel models in machine translation (Brown et al., 1993; Koehn et al., 2003; Yu et al., 2017; Yee et al., 2019) and their extensions to other tasks (Yogatama et al., 2017; Lewis and Fan, 2018).”
Toward Robust Neural Machine Translation for Noisy Input Sequences (2017, background): “Neural sequence-to-sequence models are known to suffer from explaining-away effects, where models may learn to generate outputs by relying on the target-side context while ignoring the source-side context [10, 11], especially when the source side provides only a weak or noisy signal.”
Better Document-Level Machine Translation with Bayes’ Rule (2019, methodology): “Rather than using a specially designed channel model (Yu et al., 2016) which is limited in processing long sequences like documents, our conditional sentence independence assumptions allow us to use any sequenceto-sequence model as the channel model, making it a better option for document-level…”
Simple and Effective Noisy Channel Modeling for Neural Machine Translation (2019, methodology): “trates the advantage over incremental architectures (Yu et al., 2017) that do not match vanilla seq2seq models by themselves.”
Encoder-Agnostic Adaptation for Conditional Language Generation (2019, methodology): “Recent adaptations of these ideas include the Neural Noisy Channel (Yu et al., 2017) as well as “fusion” methods (Koehn et al.”
Neural Machine Translation: A Review (2019, background): “8 explicitly control for reading source tokens and writing target tokens and thereby generate monotonic hard alignments on the segment level [210, 389].”
Latent Alignment and Variational Attention (2018, unknown): “monotonicity) on the alignment distribution [83, 82, 58].”
Putting Machine Translation in Context with the Noisy Channel Model (2019, background): “The beneﬁts of strong language models estimated from nonparallel data has long been recognized in translation (Brants et al., 2007), and more recently in noisy channel approaches to translation based on neural network component distributions (Yu et al., 2017; Yee et al., 2019; Ng et al., 2019).”
Domain Robustness in Neural Machine Translation (2019, methodology): “Standard NMT systems only model p(y|x), which can lead to a “failure mode that can occur in conditional models in which inputs are explained away by highly predictive output prefixes” (Yu et al., 2017, 1).”
Benchmarking Hierarchical Script Knowledge (2019, methodology): “To prevent the creation of a language model we force the last output word to be aligned to the last word in the abstract sequence, similar to Yu et al. (2017).”
Towards Neural Phrase-based Machine Translation (2017, background): “Another line of related work is the segment-to-segment neural transduction model (SSNT) (Yu et al., 2016), which shows promising results on a Chinese-to-English translation task under a noisy channel framework (Yu et al., 2017).”
Initial investigation of an encoder-decoder end-to-end TTS framework using marginalization of monotonic hard latent alignments (2019, methodology): “The constrained alignment is conceptually borrowed from segment-to-segment neural transduction (SSNT) [12, 13] and is extended to continuous outputs 1 .”