Parallel Iterative Edit Models for Local Sequence Transduction

By Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, Vihari Piratla. 2019

160 citations recorded by the source corpus.

We present a Parallel Iterative Edit (PIE) model for the problem of local sequence transduction arising in tasks like Grammatical error correction (GEC). Recent approaches are based on the popular encoder-decoder (ED) model for sequence to sequence learning. The ED model auto-regressively captures full dependency among output tokens but is slow due to sequential decoding. The PIE model does parallel decoding, giving up the advantage of modeling full dependency in the output, yet it achieves accuracy competitive with the ED model for four reasons: 1. predicting edits instead of tokens, 2. labeling sequences instead of generating sequences, 3. iteratively refining predictions to capture dependencies, and 4. factorizing logits over edits and their token argument to harness pre-trained language models like BERT. Experiments on tasks spanning GEC, OCR correction and spell correction demonstrate that the PIE model is an accurate and significantly faster alternative for local sequence transduction.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

PEER: A Collaborative Language Model (2022, methodology): “, 2021), fixing grammar errors (Napoles et al., 2017; Awasthi et al., 2019) or improving citations (Petroni et al.”
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020, methodology): “Kantor et al. (2019) and Awasthi et al. (2019) proposed sequence labeling models based on correction meth-ods.”
Seq2Edits: Sequence Transduction Using Span-level Edit Operations (2020, methodology): “This is very similar to the iterative decoding strategies used by Lichtarge et al. (2019); Awasthi et al. (2019); Ge et al. (2018); Grundkiewicz et al. (2019) with the difference that we pass through n-best lists between beam search passes rather than only the single best hypothesis.”
Generating bug-fixes using pretrained transformers (2021, unknown): “(Awasthi et al., 2019) pursue an encoder-only approach, which maps token zi to the edit operation to perform at the ith location.”
Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020, unknown): “One branch of the work is language-dependent approaches, like PIE (Awasthi et al., 2019) and GECToR (Omelianchuk et al., 2020).”
ReasonBERT: Pre-trained to Reason with Distant Supervision (2021, background): “…knowledge about language and signiﬁcantly boost model performance on a wide range of downstream tasks (Liu et al., 2019a; Wang et al., 2019a,b) ranging from spell checking (Awasthi et al., 2019) to sentiment analysis (Xu et al., 2019) and semantic parsing (Rongali et al., 2020), just to name a few.”
GECToR – Grammatical Error Correction: Tag, Not Rewrite (2020, methodology): “PIE (Awasthi et al., 2019) is an iterative sequence tagging GEC system that predicts token-level edit operations.”
A Comprehensive Survey of Grammar Error Correction (2020, methodology): “Iterative correction is not as widely applied and effective, but can be also used for systems based on more approaches [69], [86].”
A Comprehensive Survey of Grammatical Error Correction (2021, methodology): “PIE defines four edit operations, Copy (C), Delete (D), Append (A) with n-gram, Replace (R) with n-gram, and various lexical transformations [4].”