Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting

By Yi Zhang, Tao Ge, Furu Wei, Ming Zhou, Xu Sun. 2019

19 citations recorded by the source corpus.

We study sequence-to-sequence (seq2seq) pre-training with data augmentation for sentence rewriting. Instead of training a seq2seq model with gold training data and augmented data simultaneously, we separate them to train in different phases: pre-training with the augmented data and fine-tuning with the gold data. We also introduce multiple data augmentation methods to help model pre-training for sentence rewriting. We evaluate our approach in two typical well-defined sentence rewriting tasks: Grammatical Error Correction (GEC) and Formality Style Transfer (FST). Experiments demonstrate our approach can better utilize augmented data without hurting the model's trust in gold data and further improve the model's performance with our proposed data augmentation methods. Our approach substantially advances the state-of-the-art results in well-recognized sentence rewriting benchmarks over both GEC and FST. Specifically, it pushes the CoNLL-2014 benchmark's $F_{0.5}$ score and JFLEG Test GLEU score to 62.61 and 63.54 in the restricted training setting, 66.77 and 65.22 respectively in the unrestricted setting, and advances GYAFC benchmark's BLEU to 74.24 (2.23 absolute improvement) in E&M domain and 77.97 (2.64 absolute improvement) in F&R domain.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Eduard H. Hovy
3. Tao Ge
4. Varun Prashant Gangal
5. Soroush Vosoughi
6. Jason Wei
7. Teruko Mitamura
8. Mohit Bansal

In Natural Language Processing

1. Furu Wei
2. Varun Prashant Gangal
3. Eduard H. Hovy
4. Teruko Mitamura
5. Soroush Vosoughi
6. Tao Ge
7. Xin Sun
8. Jason Wei

In Prompting & Reasoning

1. Furu Wei
2. Tao Ge
3. Jason Wei
4. Eduard H. Hovy
5. Denny Zhou

Citations in the corpus

• A Survey of Data Augmentation Approaches for NLP (2021, methodology): “Zhang et al. (2019b) explore multi-task transfer, or the use of annotated data from other tasks.”
• Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021, methodology): “To compare with the state-of-the-art approaches in English GEC that pretrain with synthetic data, we also synthesize 300M error-corrected sentence pairs for pretraining the English GEC model following the approaches of Grundkiewicz et al. (2019) and Zhang et al. (2019).”
• Lossless Acceleration for Seq2seq Generation with Aggressive Decoding (2022, methodology): “To compare with the state-of-the-art approaches in English GEC that pretrain with synthetic data, we also synthesize 300M error-corrected sentence pairs for pretraining the English GEC model following the approaches of Grundkiewicz et al. (2019) and Zhang et al. (2019).”