ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations

By Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoît Sagot, Lucia Specia. 2020

192 citations recorded by the source corpus.

In order to simplify a sentence, human editors perform multiple rewriting transformations: they split it into several shorter sentences, paraphrase words (i.e. replacing complex words or phrases by simpler synonyms), reorder components, and/or delete information deemed unnecessary. Despite these varied range of possible text alterations, current models for automatic sentence simplification are evaluated using datasets that are focused on a single transformation, such as lexical paraphrasing or splitting. This makes it impossible to understand the ability of simplification models in more realistic settings. To alleviate this limitation, this paper introduces ASSET, a new dataset for assessing sentence simplification in English. ASSET is a crowdsourced multi-reference corpus where each simplification was produced by executing several rewriting transformations. Through quantitative and qualitative experiments, we show that simplifications in ASSET are better at capturing characteristics of simplicity when compared to other standard evaluation datasets for the task. Furthermore, we motivate the need for developing better methods for automatic evaluation using ASSET, since we show that current popular metrics may not be suitable when multiple simplification transformations are performed.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “WikiAuto + Turk/ASSET (Jiang et al., 2020; Xu et al., 2016; Alva-Manchego et al., 2020) 30.”
PEER: A Collaborative Language Model (2022, background): “…JFLEG (Napoles et al., 2017) is a grammatical error correction dataset with singlesentence inputs written by English language learners; • ASSET (Alva-Manchego et al., 2020) is a crowdsourced corpus for single-sentence text simplification; • ITERATER (Du et al., 2022b) is an editing dataset…”
Fine-tuned Language Models are Continual Learners (2022, methodology): “Moreover, CT0 obtains a new state-of-the-art on the ASSET evaluation set, improving over MUSS (Martin et al., 2020): 85.9 BLEU4 Vs 72.98 and 46.6 SARI Vs 44.15, and despite not using all the training data available.”
Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (2024, unknown): “For text simplification, which aims to simplify the source text while preserving its original meaning, we employ the ASSET dataset (Alva-Manchego et al., 2020), a benchmark known for its multiple reference translations.”
Control Prefixes for Text Generation (2021, methodology): “The GEM datasets used are DART, E2E Clean, ASSET, TurkCorpus and WebNLG+ 2020.”
Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards (2021, methodology): “The completed data card for the ASSET dataset (Alva-Manchego et al., 2020) is available in Appendix B.”
Continual-T0: Progressively Instructing 50+ Tasks to Language Models Without Forgetting (2022, methodology): “In addition, we also evaluate our models on a second Text Simpliﬁcation dataset, AS-SET (Alva-Manchego et al., 2020).”
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets (2021, background): “[4] demonstrate the importance of curating NLG datasets with multiple references when trying to train models to perform generation tasks with many acceptable outputs.”
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning (2023, unknown): “1) Text Simpliﬁcation (Simp): [23], [24] requires paraphrasing the text with a simple text; 2) Empathetic Dialogue Generation (Emdg): [25], requires the model to generate a reason for a conversational context under a given emotional situation; 3) Inquisitive Question Generation (InqQG): [26]…”
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets (2021, background): “Languages Communicative Goal Reference Data-to-text CommonGen CG en Expand concept words into natural language [32] Czech Restaurant CR cs Verbalize an agents response from dialog act [33] E2E E2 en Describe a restaurant from key-value attributes [34] ToTTo TO en Describe highlighted cells in a tabl”
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases (2022, unknown): “English We evaluate our English models on ASSET (Alva-Manchego et al., 2020a), TurkCorpus (Xu et al., 2016) and Newsela (Xu et al., 2015).”