Generating Sentences by Editing Prototypes

By Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, Percy Liang. 2017

326 citations recorded by the source corpus.

We propose a new generative language model for sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence. Compared to traditional language models that generate from scratch either left-to-right or by first sampling a latent sentence vector, our prototype-then-edit model improves perplexity on language modeling and generates higher quality outputs according to human evaluation. Furthermore, the model gives rise to a latent edit vector that captures interpretable semantics such as sentence similarity and sentence-level analogies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Such tasks include open-domain question answering [5, 25], fact checking [50], fact completion [42], long-form question answering [12], Wikipedia article generation [32], dialogue [36, 59, 9, 13], translation [16], and language modeling [17, 23].”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “Retrieve-and-edit with learned retrieval In order to better explain the variance in the input text and enable controllable generation, Guu et al. (2018) proposed a language model with the retrieve-and-edit framework (Hashimoto et al., 2018) that conditions on text with high lexical overlap.”
Generalization through Memorization: Nearest Neighbor Language Models (2019, background): “Guu et al. (2018) propose a model that samples training sentences at random and edits them with a sequence-to-sequence model, but does not use a retrieval mechanism such as k NN.”
A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, methodology): “However, the desired output might be a variation of another, previously-observed example [14, 13, 30, 18, 24].”
Retrieve and Refine: Improved Sequence Generation Models For Dialogue (2018, background): “…(Junczys-Dowmunt and Grundkiewicz, 2017; Niehues et al., 2016; Novak et al., 2016; Xia et al., 2017; Grangier and Auli, 2017) or combining with retrieval (Gu et al., 2017; Cao et al., 2018), as well as for sentence correction or refinement without context (Guu et al., 2017; Schmaltz et al., 2017).”
Neural Text Generation with Unlikelihood Training (2019, background): “Some other lines of work of note are the use of generative control (Kikuchi et al., 2016; Fan et al., 2017; Peng et al., 2018; See et al., 2019) and the retrieve-and-reﬁne setting (Weston et al., 2018; Guu et al., 2018).”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, methodology): “Prompt augmentation is closely related to retrieval-based methods that provide more textual context to the model to improve performance [37], a method that has also been shown to be effective in prompt-based learning [99].”
What Makes Good In-Context Examples for GPT-3? (2021, background): “Some representative applications in the ﬁeld of deep learning include machine translation (Gu et al., 2018), sentiment transfer (Li et al., 2018; Guu et al., 2018), QA (Karpukhin et al., 2020; Mao et al., 2020), dialogue generation (Yan et al., 2016; Cai et al., 2018; Song et al., 2016; Pandey et…”
Foundation Models and Fair Use (2023, methodology): “It’s worth noting that retrieval-augmented methods Guu et al. (2018, 2020), which perform attribution fundamentally and not post-hoc, are another approach to instance attribution.”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, methodology): “The technique of editing exemplar sentences has been used in semiparametric generation models (Guu et al., 2018; Pandey et al., 2018; Peng et al., 2019) and crowd-sourcing small, iterative changes to text has been shown to lead to higher-quality data and a more robust annotation process (Little et…”
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks (2018, background): “Shen et al. (2017) perform style transfer on non-parallel texts, while Guu et al. (2017) generate novel sentences from prototypes; again, these methods are not necessarily seeking to generate meaning-preserving paraphrases, merely transformed sentences that have an altered style.”
PAWS: Paraphrase Adversaries from Word Scrambling (2019, background): “Most text generation models rely on large amount of training data (Iyyer et al., 2018; Guu et al., 2018; Gupta et al., 2018; Li et al., 2018), which is unfortunately not available in our case.”
Delete, Retrieve, Generate: a Simple Approach to Sentiment and Style Transfer (2018, methodology): “For D ELETE A ND R ETRIEVE , similar to Guu et al. (2018), we retrieve the top-10 sentences and generate results using markers from each sentence.”
Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization (2018, methodology): “Guu et al. (2017) also proposed to encode human-written sentences to improvement the performance of neural text generation.”