Incorporating Copying Mechanism in Sequence-to-Sequence Learning

By Jiatao Gu, Zhengdong Lu, Hang Li, Victor O. K. Li. 2016

1,593 citations recorded by the source corpus.

We address an important problem in sequence-to-sequence (Seq2Seq) learning referred to as copying, in which certain segments in the input sequence are selectively replicated in the output sequence. A similar phenomenon is observable in human language communication. For example, humans tend to repeat entity names or even long phrases in conversation. The challenge with regard to copying in Seq2Seq is that new machinery is needed to decide when to perform the operation. In this paper, we incorporate copying into neural network-based Seq2Seq learning and propose a new model called CopyNet with encoder-decoder structure. CopyNet can nicely integrate the regular way of word generation in the decoder with the new copying mechanism which can choose sub-sequences in the input sequence and put them at proper places in the output sequence. Our empirical study on both synthetic data sets and real world data sets demonstrates the efficacy of CopyNet. For example, CopyNet can outperform regular RNN-based model with remarkable margins on text summarization tasks.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

CoQA: A Conversational Question Answering Challenge (2018, methodology): “Additionally, as the answer words are likely to appear in the original passage, we employ a copy mechanism in the decoder which allows to (optionally) copy a word from the passage (Gu et al., 2016; See et al., 2017).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “…set of baselines for decaNLP that combine the basics of sequence-to-sequence learning [Sutskever et al., 2014, Bahdanau et al., 2014, Luong et al., 2015b] with pointer networks [Vinyals et al., 2015, Merity et al., 2017, Gülçehre et al., 2016, Gu et al., 2016, Nallapati et al., 2016], ad- Preprint.”
Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “Our approach is close to the Forced-Attention Sentence Compression model of Miao and Blun-som (2016) and the CopyNet model of Gu et al. (2016), with some small differences: (i) We calculate an explicit switch probability p gen , whereas Gu et al. induce competition through a shared soft-max…”
Neural Generation for Czech: Data and Baselines (2019, background): “Delexicalization or an analogous technique, such as a copy mechanism (Gu et al., 2016; Gehrmann et al., 2018), is required for most generation scenarios to allow generalization to unseen entity names: sets of entities are open (potentially infinite and subject to change) while training data is scarc”
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, methodology): “Furthermore, we apply a “pointer network” (Gu et al., 2016) to enable the model to copy words from input.”
A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, methodology): “In our experiments, we let pedit be a standard sequence-to-sequence model with attention and copying [12, 36] (see Appendix B for details), but any model architecture can be used for the editor.”
Break It Down: A Question Understanding Benchmark (2020, methodology): “We use BREAK to train a neural QDMR parser that maps questions into QDMR representations, based on a sequence-to-sequence model with copying (Gu et al., 2016).”
Towards VQA Models That Can Read (2019, methodology): “To generalize to arbitrary text, we take inspiration from pointer networks which allow pointing to OOV words in context [11, 39, 32, 12, 34].”
On Faithfulness and Factuality in Abstractive Summarization (2020, methodology): “…coherence, mostly thanks to advances in sequenceto-sequence architectures with attention and copy (Sutskever et al., 2014; Bahdanau et al., 2015; Gu et al., 2016), fully attention-based Transformer architectures (Vaswani et al., 2017; Dai et al., 2019) and more recently pretrained language…”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, methodology): “We implement the generative model as a sequence-to-sequence model (Sutskever et al., 2014) with a copy mechanism (Gu et al., 2016; Gulcehre et al., 2016).”
Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning (2017, methodology): “Pointer models have also been successfully applied to tasks such as language modeling (Merity et al., 2017), summarization (Gu et al., 2016), combinatorial optimization (Bello et al., 2017), and question answering (Seo et al., 2017; Xiong et al., 2017).”
Neural Approaches to Conversational AI (2018, positive): “An instance of this model is CopyNet (Gu et al., 2016), which was shown to significantly improve over RNNs thanks to its ability to repeat proper nouns and other words of the input.”
Bottom-Up Abstractive Summarization (2018, methodology): “Current state-of-the-art neural abstractive summarization models combine extractive and abstractive techniques by using pointergenerator style models which can copy words from the source document (Gu et al., 2016; See et al., 2017).”
Challenges in Data-to-Document Generation (2017, background): “Copying There has been a surge of recent work involving augmenting encoder-decoder models to copy words directly from the source material on which they condition (Gu et al., 2016; G¨ulc¸ehre et al., 2016; Merity et al., 2016; Jia and Liang, 2016; Yang et al., 2016).”
Text Summarization with Pretrained Encoders (2019, background): “…encoding for the summarization task; a variety of recently proposed techniques aim to enhance summarization performance via copying mechanisms (Gu et al., 2016; See et al., 2017; Nallap-ati et al., 2017), reinforcement learning (Narayan et al., 2018b; Paulus et al., 2018; Dong et al., 2018),…”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, methodology): “We use them with the addition of attention mecha-2 Relation deﬁnitions are at https://github.com/ commonsense/conceptnet5/wiki/Relations . nism (Luong et al., 2015) with copying ability (Gu et al., 2016), which are based on an open-source framework OpenNMT-py (Klein et al., 2017).”
Pointing the Unknown Words (2016, methodology): “To make the pointing mechanism applicable in more general settings, our model learns two key abilities jointly: (i) to predict whether it is required to use the pointing or not at each time step and (ii) to point any location of the context sequence whose length can vary widely over examples.”
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents (2018, background): “…There has been a surge of recent works in sequence learning tasks to address the problem of unkown token prediction by allowing the model to occasionally copy words directly from source instead of generating a new token (Gu et al., 2016; See et al., 2017; Paulus et al., 2017; Wiseman et al., 2017).”