Generating Wikipedia by Summarizing Long Sequences

By Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, Noam Shazeer. 2018

875 citations recorded by the source corpus.

We show that generating English Wikipedia articles can be approached as a multi- document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder- decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores and human evaluations.

Subfields: Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Generative Models

Citations in the corpus

Scaling Laws for Neural Language Models (2020, methodology): “In this work we will empirically investigate the dependence of language modeling loss on all of these factors, focusing on the Transformer architecture [VSP + 17, LSP + 18].”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Such tasks include open-domain question answering [5, 25], fact checking [50], fact completion [42], long-form question answering [12], Wikipedia article generation [32], dialogue [36, 59, 9, 13], translation [16], and language modeling [17, 23].”
Learning Transferable Visual Models From Natural Language Supervision (2021, background): “To our knowledge Liu et al. (2018) first identified task learning as an “unexpected side-effect” when a language model trained to generate Wikipedia articles learned to reliably transliterate names between languages.”
Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, background): “, 2019), and Transformer-DMCA (Liu et al., 2018) are a few examples of large, publicly available models with this ability.”
Generating Long Sequences with Sparse Transformers (2019, methodology): “A simple and powerful choice for model θ is a Transformer (Vaswani et al., 2017) in decoder-only mode, as demonstrated by (Radford et al., 2018) and (Liu et al., 2018).”
Music Transformer: Generating Music with Long-Term Structure (2018, methodology): “With a larger dataset, local attention may fare well as it allows for much deeper models and longer sequences, as seen in text and image generation work (Liu et al., 2018; Parmar et al., 2018)).”
Improving Question Answering with External Knowledge (2019, background): “ recent studies (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018) leverage rich world knowledge by pre-training deep neural models such as LSTMs and Transformers (Vaswani et al., 2017; Liu et al., 2018) using language model objectives over large-scale corpora (e.g., BookCorpus (Zhu et”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “WikiSum (Liu et al., 2018) 32.”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, background): “Later, hierarchical attention has been simpli-ﬁed by (Liu et al., 2018) that alternates coarse layers (attending to the whole sequence at a lower temporal resolution) with local layers (attending to a neighborhood of the current prediction).”
Reformer: The Efficient Transformer (2020, background): “Up to 11 thousand tokens of text in a single example were processed in (Liu et al., 2018) and when processing other modalities, like music (Huang et al., 2018) and images (Parmar et al., 2018), even longer sequences are commonplace.”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, background): “Since average input length in BIGPATENT, arXiv, PubMed and Multi-News are well beyond 1024 tokens, further scaling up Linput or applying a two-stage approach (Liu et al., 2018) may improve performance even more, although this is outside the scope of this work.”
Fast Transformer Decoding: One Write-Head is All You Need (2019, methodology): “Another is to reduce the number of positions being attended-to, either by attending to a local neighborhood, or by otherwise compressing the number of memory positions, as in [Liu et al., 2018], [Zhang et al., 2018], [Povey et al., 2018].”
A General Language Assistant as a Laboratory for Alignment (2021, methodology): “All the decoder-only [LSP18] Transformer [VSP17] models we train have a fixed aspect ratio dmodel/nlayer = 128, as it has been shown that this is roughly optimal [KMH20].”
ELI5: Long Form Question Answering (2019, background): “WikiSum (Liu et al., 2018) proposes writing Wikipedia articles as a multi-document summarization task.”
Hierarchical Neural Story Generation (2018, neutral): “Recent work has applied these models to more open-ended generation tasks, including writing Wikipedia articles (Liu et al., 2018) and poetry (Zhang and Lapata, 2014).”
Improving Wikipedia verifiability with AI (2022, background): “Several works have investigated the ability of AI to generate missing Wikipedia articles from scratch (Liu et al., 2018; Prabhumoye et al., 2019; Fan & Gardent, 2022; Kaffee et al., 2022).”
Hindsight: Posterior-guided training of retrievers for improved open-ended generation (2021, background): “, 2019), or Wikipedia abstract generation (Liu* et al., 2018), many responses can be equally acceptable and yet the data source (e.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “ord et al., 2018], another model from which it traces intellectual ancestry. If BERT can be viewed as an encoder-only transformer, GPT is the exact opposite: it represents a decoder-only transformer [Liu et al., 2018a], or the “decoder half” of a full sequence-to-sequence transformer model. GPT is p”