Optimizing Statistical Machine Translation for Text Simplification

By Wei Xu, Courtney Napoles, Ellie Pavlick, Quan Ze Chen, Chris Callison-Burch. 2016

757 citations recorded by the source corpus.

Most recent sentence simplification systems use basic machine translation models to learn lexical and syntactic paraphrases from a manually simplified parallel corpus. These methods are limited by the quality and quantity of manually simplified corpora, which are expensive to build. In this paper, we conduct an in-depth adaptation of statistical machine translation to perform text simplification, taking advantage of large-scale paraphrases learned from bilingual texts and a small amount of manual simplifications with multiple references. Our work is the first to design automatic metrics that are effective for tuning and evaluating simplification systems, which will facilitate iterative development for this task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “WikiAuto + Turk/ASSET (Jiang et al., 2020; Xu et al., 2016; Alva-Manchego et al., 2020) 30.”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, neutral): “– GEM (en): Wiki-Lingua (Faisal Ladhak & McKeown, 2020), WenNLG (Gardent et al., 2017; Castro Ferreira et al., 2020), CommonGEN (Lin et al., 2020), E2E (Dušek et al., 2019), DART (Radev et al., 2020), ToTTo (Parikh et al., 2020), Wiki-Auto (Jiang et al., 2020), TurkCorpus(Xu et al., 2016) – DialoGL”
PEER: A Collaborative Language Model (2022, background): “PEER substantially outperforms all other models in this group, with the 3B model achieving an average SARI score of 42.0 across all tasks, compared to 24.1 for the strongest T5based baseline model.”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, neutral): “Similar to SARI for text simplification (Xu et al., 2016) and Q-BLEU for question generation (Nema and Khapra, 2018), PARENT falls under the category of task-specific metrics.”
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2019, neutral): “(Xu et al., 2016) and the exact match accuracy.”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “The SARI KEEP F1 (Xu et al., 2016) of 75 shows that the model and the reference mostly agree on parts of the sentence that should be kept unchanged.”
Neural CRF Model for Sentence Alignment in Text Simplification (2020, positive): “We used the benchmark T URK corpus (Xu et al., 2016) for evaluation on Wikipedia, which consists of 8 human-written references for sentences in the validation and test sets.”
Chain of Explanation: New Prompting Method to Generate Quality Natural Language Explanation for Implicit Hate Speech (2022, methodology): “The SARI metric measures the goodness of words by comparing generated NLE with the ground truth NLE [26].”
BLEU is Not Suitable for the Evaluation of Text Simplification (2018, methodology): “In one (“Standard Reference Setting”, § 4.2), we use two sets of references: the Simple Wikipedia reference (yielding BLEU-1ref and iBLEU-1ref), and 8 references obtained by crowdsourcing by Xu et al. (2016) (yielding BLEU-8ref, iBLEU-8ref and SARI-8ref).”
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021, neutral): “We report corpus-level BLEU14, the exact match accuracy, and SARI scores (Xu et al., 2016)15.”
FELIX: Flexible Text Editing Through Tagging and Insertion (2020, methodology): “Following Geva et al. (2019), we report two metrics: Exact score , which is the percentage of exactly correctly predicted fusions, and SARI (Xu et al., 2016), which computes the average F1 scores of the added, kept, and deleted n-grams.”
Understanding Iterative Revision from Human-Written Text (2022, methodology): “FKGL is a popular metric that has been used by many prior works (Solnyshkina et al., 2017; Xu et al., 2016; Guo et al., 2018; Nassar et al., 2019; Nishihara et al., 2019) to measure the read-ability of documents.”
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases (2022, neutral): “…and unsupervised simplification systems can be adapted to different conditions: (Kumar et al., 2020) choose operation-specific weights of their unsupervised simplification model for each benchmark and (Surya et al., 2019) select different models using SARI (Xu et al., 2016) on each validation set.”
Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer (2018, methodology): “Phrased-based machine translation models have had success in the ﬁelds of machine translation, style transfer (X U 12) and text simpliﬁcation (Wubben et al., 2012b; Xu et al., 2016).”
UvA-DARE (Digital Academic Repository) A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs (neutral): “A similar insight was used in one of the component metrics of the SARI evaluation metric used for text simplification evaluation (Xu et al., 2016).”
Evaluating Factuality in Text Simplification (2022, methodology): “As Table 7 reports, there is only a weak correlation between SARI and the prevalence of information errors, and both the direction and magnitude of the correlation are highly dependent on model and dataset.”
Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework (2021, methodology): “(A2) and (S1) are the simpler version of (A1) and (S2) (Xu et al., 2016).”