Encode, Tag, Realize: High-Precision Text Editing
By Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, Aliaksei Severyn. 2019
191 citations recorded by the source corpus.
We propose LaserTagger - a sequence tagging approach that casts text generation as a text editing task. Target texts are reconstructed from the inputs using three main edit operations: keeping a token, deleting it, and adding a phrase before the token. To predict the edit operations, we propose a novel model, which combines a BERT encoder with an autoregressive Transformer decoder. This approach is evaluated on English text on four tasks: sentence fusion, sentence splitting, abstractive summarization, and grammar correction. LaserTagger achieves new state-of-the-art results on three of these tasks, performs comparably to a set of strong seq2seq baselines with a large number of training examples, and outperforms them when the number of examples is limited. Furthermore, we show that at inference time tagging can be more than two orders of magnitude faster than comparable seq2seq models, making it more attractive for running in a live environment.
Subfields: Artificial Intelligence, Natural Language Processing, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Furu Wei
- Graham Neubig
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Felix Stahlberg
- Jianfeng Gao
- Uri Alon
In Natural Language Processing
In Natural Language Generation
Citations in the corpus
- Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021, methodology): “LaserTagger (Malmi et al., 2019) uses the similar method but it is data-driven and language-independent by learning operations from training data.”
- NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction (2021, methodology): “…to recent work on casting text generation (e.g. summarization, grammatical error correction, sentence splitting, etc.) as a text editing task (Malmi et al., 2019; Panthaplackel et al., 2020; Stahlberg and Kumar, 2020) where target texts are reconstructed from inputs using several edit…”
- A Comprehensive Survey of Grammar Error Correction (2020, methodology): “Unlike the commonly used ED model, the PIE model generates the output concurrently, much faster than the encoder-decoder model.”
- Understanding Iterative Revision from Human-Written Text (2022, methodology): “We choose these automatic metrics following prior text revision works (Malmi et al., 2019; Dong et al., 2019; Mallinson et al., 2020).”
- Learning to Model Editing Processes (2022, background): “Other work on editing has included editing for sentence fusion (Malmi et al., 2019), in which one can perform minimal operations to join two sentences together grammati-cally.”
- Learning to Generate Code Sketches (2021, background): “While LMs generate realistic-looking outputs, they are known to occasionally “hallucinate” [27, 22, 23, 19], i.”
- Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2022, result): “…(Ge et al., 2022b; Yang et al., 2023) present extensive empirical studies of Speculative Decoding in special cases (e.g., text editing or retrieval-augmented generation) and compare it with competitive approaches (Malmi et al., 2019; Omelianchuk et al., 2020; Chen et al., 2020) in those scenarios.”
- AgreeSum: Agreement-Oriented Multi-Document Summarization (2021, background): “More broadly, there have been other recent works striving to develop techniques for high-precision generation (Malmi et al., 2019; Tian et al., 2019; Pang and He, 2020; Parikh et al., 2020; Dušek and Kasner, 2020).”
- A Comprehensive Survey of Grammatical Error Correction (2021, unknown): “Editing operations in LaserTagger consist of two parts, a basic operation and an append phrase P operation [88].”
- Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020, unknown): “Although the non auto-regressive models like Levenshtein Transformer and LaserTagger are faster than the seq2seq baseline, their performance is not desirable.”
- FELIX: Flexible Text Editing Through Tagging and Insertion (2020, methodology): “Based on this observation, text-editing approaches have recently re-gained significant interest (Gu et al., 2019; Dong et al., 2019; Awasthi et al., 2019; Malmi et al., 2019).”
- GECToR – Grammatical Error Correction: Tag, Not Rewrite (2020, methodology): “LaserTagger (Malmi et al., 2019) combines a BERT encoder with an autoregressive Transformer decoder to predict three main edit operations: keeping a token, deleting a token, and adding a phrase before a token.”
- Seq2Edits: Sequence Transduction Using Span-level Edit Operations (2020, methodology): “A more recent line of research frames sequence editing as a labelling problem using labels such as ADD, KEEP, and DELETE (Ribeiro et al., 2018; Dong et al., 2019; Mallinson et al., 2020; Malmi et al., 2019; Awasthi et al., 2019), often relying heavily on BERT (Devlin et al., 2019) pre-training.”
- Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021, methodology): “Text editing models (Malmi et al., 2019; Dong et al., 2019b; Mallinson et al., 2020) cast text generation as a sequence tagging problem with carefully selected edit operations required for the task.”
- Neural Edit Completion (2020, methodology): “SequenceR LSTM [Chen et al. 2019] +copy 30.7 10M SequenceR Transformer [Chen et al. 2019] +copy 32.6 45M LaserTagger LSTM [Malmi et al. 2019] +CRF 40.9 1M LaserTagger Transformer [Malmi et al. 2019] +CRF 41.4 1.6M Path2Tree LSTM [Aharoni and Goldberg 2017] Table 2.”