Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data

By Roman Grundkiewicz, Marcin Junczys-Dowmunt, Kenneth Heafield. 2019

191 citations recorded by the source corpus.

Considerable effort has been made to address the data sparsity problem in neural grammatical error correction. In this work, we propose a simple and surprisingly effective unsupervised synthetic error generation method based on confusion sets extracted from a spellchecker to increase the amount of training data. Synthetic data is used to pre-train a Transformer sequence-to-sequence model, which not only improves over a strong baseline trained on authentic error-annotated data, but also enables the development of a practical GEC system in a scenario where little genuine error-annotated data is available. The developed systems placed first in the BEA19 shared task, achieving 69.47 and 64.24 F_{0.5} in the restricted and low-resource tracks respectively, both on the W&I+LOCNESS test set. On the popular CoNLL 2014 test set, we report state-of-the-art results of 64.16 M² for the submitted system, and 61.30 M² for the constrained system trained on the NUCLE and Lang-8 data.

Subfields: Artificial Intelligence, Natural Language Processing, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Furu Wei
  2. Tao Ge
  3. Felix Stahlberg
  4. Shankar Kumar
  5. M. Zhou
  6. Xin Sun
  7. Jianfeng Gao
  8. Mariano Felice

In Natural Language Processing

  1. Furu Wei
  2. Tao Ge
  3. Felix Stahlberg
  4. Shankar Kumar
  5. Xin Sun
  6. M. Zhou
  7. Mariano Felice
  8. Zhuowei Liu

In Natural Language Generation

  1. Furu Wei
  2. Tao Ge
  3. Shankar Kumar
  4. Felix Stahlberg
  5. Xin Sun
  6. Houfeng Wang
  7. Heming Xia
  8. Zhuowei Liu

Citations in the corpus