Efficient Summarization with Read-Again and Copy Mechanism

By Wenyuan Zeng, Wenjie Luo, Sanja Fidler, Raquel Urtasun. 2016

101 citations recorded by the source corpus.

Encoder-decoder models have been widely used to solve sequence to sequence prediction tasks. However current approaches suffer from two shortcomings. First, the encoders compute a representation of each word taking into account only the history of the words it has read so far, yielding suboptimal representations. Second, current models utilize large vocabularies in order to minimize the problem of unknown words, resulting in slow decoding times and large storage costs. In this paper we address both shortcomings. Towards this goal, we first introduce a simple mechanism that first reads the input sequence before committing to a representation of each word. Furthermore, we propose a simple copy mechanism that is able to exploit very small vocabularies and handle out-of-vocabulary words. We demonstrate the effectiveness of our approach on the Gigaword dataset and DUC competition outperforming the state-of-the-art.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “For example, the summaries on the DUC-2004 dataset generated by the state-of-the-art sys-tem by Zeng et al. (2016) are limited to 75 characters.”
Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “The pointer network has been used to create hybrid approaches for NMT (Gul-cehre et al., 2016), language modeling (Merity et al., 2016), and summarization (Gu et al., 2016; Gulcehre et al., 2016; Miao and Blunsom, 2016; Nallapati et al., 2016; Zeng et al., 2016).”
Bottom-Up Abstractive Summarization (2018, background): “Finally Zeng et al. (2016) introduce a mechanism that reads a source document in two passes and uses the information from the ﬁrst pass to bias the second.”
Deep Communicating Agents for Abstractive Summarization (2018, background): “Attention mechanisms have shown to be crucial for summarization as well (Rush et al., 2015; Zeng et al., 2016; Nallapati et al., 2017), and pointer networks (Vinyals et al., 2015a), in particular, help address redundancy and saliency in generated summaries (Cheng and Lapata, 2016; See et al., 2017;…”
Ensure the Correctness of the Summary: Incorporate Entailment Knowledge into Abstractive Sentence Summarization (2018, background): “Gu et al. (2016) and Zeng et al. (2016) incorporate a copying mechanism into seq2seq learning and Gulcehre et al. (2016) propose a switch gate to control whether to copy from the source or generate a word by the decoder.”
Deep Keyphrase Generation (2017, background): “In order to make use of the important information in the source text, some studies sought ways to copy certain parts of content from the source text and paste them into the target text (Allamanis et al., 2016; Gu et al., 2016; Zeng et al., 2016).”
Topic-to-Essay Generation with Neural Networks (2018, methodology): “As for summarization [Zeng et al. , 2016], it condenses one long article to several sentences (usually less than 5) to highly summarize the article.”
Selective Encoding for Abstractive Sentence Summarization (2017, methodology): “Zeng et al. (2016) also propose using copy mechanism and add a scalar weight on the gate of GRU/LSTM for this task. Cheng and Lapata (2016) use an RNN based encoder-decoder for extractive summarization of documents.”
Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models (2017, methodology): “, 2017) (Zeng et al., 2016) and question answering (Wang and Jiang, 2016).”