Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model

By Kyunghyun Cho. 2016

66 citations recorded by the source corpus.

Recent advances in conditional recurrent language modelling have mainly focused on network architectures (e.g., attention mechanism), learning algorithms (e.g., scheduled sampling and sequence-level training) and novel applications (e.g., image/video description generation, speech recognition, etc.) On the other hand, we notice that decoding algorithms/strategies have not been investigated as much, and it has become standard to use greedy or beam search. In this paper, we propose a novel decoding strategy motivated by an earlier observation that nonlinear hidden layers of a deep neural network stretch the data manifold. The proposed strategy is embarrassingly parallelizable without any communication overhead, while improving an existing decoding algorithm. We extensively evaluate it with attention-based neural machine translation on the task of En→Cz translation.

Subfields: Artificial Intelligence, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Generative Models

Citations in the corpus

Non-Autoregressive Neural Machine Translation (2017, background): “…the expectation of its corresponding soft-max distribution: Noisy parallel decoding (NPD) A more accurate approximation of the true optimum of the target distribution, inspired by Cho (2016), is to draw samples from the fertility space and compute the best translation for each fertility sequence.”
Neural Combinatorial Optimization with Reinforcement Learning (2016, neutral): “We also considered perturbing the pointing mechanism with random noise and greedily decoding from the obtained modified policy, similarly to (Cho, 2016), but this proves less effective than samplingin our experiments.”
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018, background): “Among these, beam search has become the method of choice, due to its superior performance over greedy decoding, which however comes with a substantial computational overhead (Cho, 2016).”
A Simple, Fast Diverse Decoding Algorithm for Neural Generation (2016, methodology): “We ﬁrst go over the vanilla beam search method and then detail the proposed algorithm which fosters diversity during decoding.”
Analyzing Uncertainty in Neural Machine Translation (2018, background): “…(Blunsom et al., 2008; Zhang et al., 2016) which explicitly attempt to model multiple modes in the data distribution, or decoding strategies which attempt to predict diverse outputs while leaving the model unchanged (Gimpel et al., 2013; Vijayakumar et al., 2016; Li & Jurafsky, 2016; Cho, 2016).”
Levenshtein Transformer (2019, methodology): “Moreover, we find that using search (instead of greedy decoding) or nosiy parallel decoding (Cho, 2016) does not yield much gain in LevT.”
Towards Decoding as Continuous Optimisation in Neural Machine Translation (2017, methodology): “The most widely-used inference methods include sampling (Cho, 2016), greedy and beam search (Sutskever et al.”
A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Models (2019, methodology): “We obtain a diverse collection of samples using noise injection sampling (Cho, 2016).”
Can Unconditional Language Models Recover Arbitrary Sentences? (2019, methodology): “Unlike forward estimation, backward estimation is a combinatorial optimization problem and cannot be solved easily with a recurrent language model (Cho, 2016; Chen et al., 2018).”
A Stable and Effective Learning Strategy for Trainable Greedy Decoding (2018, methodology): “NPAD Noisy parallel approximate decoding (NPAD; Cho, 2016) is a parallel decoding algo-rithm that can be used to improve greedy decoding or beam search.”
Trainable Greedy Decoding for Neural Machine Translation (2017, methodology): “We start from the noisy, parallel approximate decoding (NPAD) algorithm proposed in (Cho, 2016).”
Decoding and Diversity in Machine Translation (2020, background): “moothing. Other studies propose decoding strategies to increase diversity, but lack comparisons to the ground truth distribution (Gimpel et al., 2013; Vijayakumar et al., 2016; Li and Jurafsky, 2016; Cho, 2016). It has been observed that NMT can often produce misgendered outputs, and solutions focus”
Comparison of Diverse Decoding Methods from Conditional Language Models (2019, methodology): “Iterative Beam Search In an attempt to improve the size of the search space explored without sacriﬁcing runtime, Kulikov et al. (2018) propose an iterative beam search method.”
Neural Machine Translation: A Review (2019, background): “Note that both extensions can be used together (Cho, 2016).”
Mixture Content Selection for Diverse Sequence Generation (2019, background): “Cho (2016) adds random noise to RNN decoder hidden states.”