Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding

By Xin Sun, Tao Ge, Furu Wei, Houfeng Wang. 2021

77 citations recorded by the source corpus.

In this paper, we propose Shallow Aggressive Decoding (SAD) to improve the online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC). SAD optimizes the online inference efficiency for GEC by two innovations: 1) it aggressively decodes as many tokens as possible in parallel instead of always decoding only one token in each step to improve computational parallelism; 2) it uses a shallow decoder instead of the conventional Transformer architecture with balanced encoder-decoder depth to reduce the computational cost during inference. Experiments in both English and Chinese GEC benchmarks show that aggressive decoding could yield identical predictions to greedy decoding but with significant speedup for online inference. Its combination with the shallow decoder could offer an even higher online inference speedup over the powerful Transformer baseline without quality loss. Not only does our approach allow a single model to achieve the state-of-the-art results in English GEC benchmarks: 66.4 F0.5 in the CoNLL-14 and 72.9 F0.5 in the BEA-19 test set with an almost 10x online inference speedup over the Transformer-big model, but also it is easily adapted to other languages. Our code is available at https://github.com/AutoTemp/Shallow-Aggressive-Decoding.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Ge
2. Furu Wei
3. Heming Xia
4. Zhifang Sui
5. Peiyi Wang
6. Qingxiu Dong
7. Tianyu Liu
8. Yuhui Li

In Natural Language Processing

1. Tao Ge
2. Furu Wei
3. Heming Xia
4. Zhifang Sui
5. Peiyi Wang
6. Tianyu Liu
7. Qingxiu Dong
8. Yuhui Li

Citations in the corpus

• Fast Inference from Transformers via Speculative Decoding (2022, neutral): “Shallow Aggressive Decoding (SAD) (Sun et al., 2021) also decodes several tokens in parallel, similarly to our work.”
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024, methodology): “Early attempts at Speculative Decoding focused on the verification criterion that supports greedy decoding, which guarantees that the outputs are exactly the same as the greedy decoding results of the target LLM (Stern et al., 2019; Sun et al., 2021; Xia et al., 2022).”
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, background): “Early works (Stern et al., 2018; Sun et al., 2021) accelerated greedy decoding, while speculative sampling (Leviathan et al., 2023; Chen et al., 2023a) extended it to non-greedy sampling, provably maintaining the original output distribution.”
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (2024, background): “Early speculative decoding methods (Stern et al., 2018; Sun et al., 2021) accelerated generation in greedy settings, while Leviathan et al. (2023); Chen et al. (2023a) proposed speculative sampling to extend the draft-verification framework to non-greedy generation.”
• Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2022, methodology): “However, previous attempts in the “ draft-then-verify ” paradigm such as Blockwise Decoding (Stern et al., 2018) and Aggressive Decoding (Sun et al., 2021) tend to lack in-depth investigation of this paradigm.”