Deliberation Networks: Sequence Generation Beyond One-Pass Decoding

By Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, Tie-Yan Liu. 2017

221 citations recorded by the source corpus.

Subfields: Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

Citations in the corpus

Microsoft Research Asia’s Systems for WMT19 (2019, methodology): “Furthermore, some other techniques such as de-liberation learning (Xia et al., 2017b), adversarial learning (Wu et al., 2018b), and reinforcement learning (He et al., 2017; Wu et al., 2018a) could also hep and are worthy of exploration.”
Retrieve and Refine: Improved Sequence Generation Models For Dialogue (2018, background): “…MT and summarization either for refinement of initial predictions (Junczys-Dowmunt and Grundkiewicz, 2017; Niehues et al., 2016; Novak et al., 2016; Xia et al., 2017; Grangier and Auli, 2017) or combining with retrieval (Gu et al., 2017; Cao et al., 2018), as well as for sentence correction or…”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, methodology): “We address this limitation in two ways: a two-pass decoding (Deliberation Networks) 3.4.1 and Agreement Regularization 3.4.2.”
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018, background): “QuickEdit (Grang-ier and Auli, 2017) and deliberation network (Xia et al., 2017) incorporate the idea of reﬁnement into neural machine translation.”
Classical Structured Prediction Losses for Sequence to Sequence Learning (2017, methodology): “If we train Risk only on the news-commentary portion of the training data, then we achieve state of the art accuracy on this dataset of 41.5 BLEU (Xia et al., 2017).”
Dual Transfer Learning for Neural Machine Translation with Marginal Distribution Regularization (2018, methodology): “Beyond the standard encoder-decoder architecture, more elaborate decoder architectures have been proposed to promote the performance of NMT systems (Xia et al. 2017c; He et al. 2017).”
Levenshtein Transformer (2019, methodology): “QuickEdit (Grangier and Auli, 2017) and deliberation network (Xia et al., 2017) both consist of two autoregressive decoders where the second decoder refines the translation generated by the first decoder.”
Response Generation by Context-aware Prototype Editing (2018, methodology): “A typical approach follows a writing-then-edit paradigm, that utilizes one decoder to generate a draft from scratch and uses another decoder to revise the draft (Xia et al. 2017).”
Depth Growing for Neural Machine Translation (2019, background): “…deep neural networks, has gained rapid progress in recent years (Bahdanau et al., 2015; Sutskever et al., 2014; Sennrich et al., 2016b; He et al., 2016a; Sennrich et al., 2016a; Xia et al., 2017; Wang et al., 2019) and achieved signiﬁcant improvement in translation quality (Has-san et al., 2018).”
A Compare Aggregate Transformer for Understanding Document-grounded Dialogue (2020, methodology): “Meanwhile, a two-way deliberation decoder (Xia et al., 2017) was used for response generation.”
Forward–Backward Decoding Sequence for Regularizing End-to-End TTS (2019, background): “sequence generation tasks, such as end-to-end speech recognition [24] and neural machine translation (NMT), [25], [26].”
Scene Text Detection and Recognition: The Deep Learning Era (2018, background): “…module that performs translations from coarse, prone-to-error text outputs from the decoder to fine and linguistically calibrated outputs,which bears some resemblance to the deliberation networks for machine translation (Xia et al. 2017) that first translate and then re-write the sentences.”
Decoding with Value Networks for Neural Machine Translation (2017, background): “Neural Machine Translation (NMT), which is based on deep neural networks and provides an endto-end solution to machine translation, has attracted much attention from the research community [2, 6, 12, 20] and gradually been adopted by industry in past several years [18, 22].”
Towards Accurate Text-based Image Captioning with Content Diversity Exploration (2021, methodology): “Inspired by the Deliberation Network [48], as shown in Figure 2, AnCM consists of a visual-captioner (denoted as AnCMv) and a text-captioner (denoted as AnCMt).”
Distilling the Knowledge of BERT for Sequence-to-Sequence ASR (2020, background): “In [20], a second-pass deliberation decoder that can leverage right context was proposed.”
Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences (2020, background): “Refinement of initial hypotheses by a secondary expert model, on the other hand, follows the general principle underlying deliberation networks initially developed to improve machine translation quality (Xia et al., 2017; Wang et al., 2019b), although limited to inference only for our purposes.”
Fluency Boost Learning and Inference for Neural Grammatical Error Correction (2018, methodology): “• CAMB14, CAMB16 SMT , CAMB16 NMT and CAMB17: GEC systems (Felice et al., 2014; Yuan et al., 2016; Yuan and Briscoe, 2016; Yannakoudakis et al., 2017) developed by Cambridge University.”
A Study of Reinforcement Learning for Neural Machine Translation (2018, background): “Recently, neural machine translation (NMT) (Bah-danau et al., 2015; Hassan et al., 2018; Wu et al., 2016; He et al., 2017; Xia et al., 2016, 2017; Wu et al., 2018b,a) has become more and more popular given its superior performance without the demand of heavily hand-crafted engineering efforts.”
Towards Enhancing Faithfulness for Neural Machine Translation (2020, background): “Then, several related researches about improve faithfulness of NMT (Kong et al., 2019; Zheng et al., 2019; Chen et al., 2019; Feng et al., 2020) or exploiting translations for improving NMT (Xia et al., 2017; Wang et al., 2019) also be reported (line 7-12).”